Let me explain the solution to two special cases of
the equational/co-equational problem.
Special subcase 1: the restriction of the problem to
abelian groups only, and
Special subcase 2: the restriction of the problem to the
special case mentioned in the problem statement.
That is, $H = \{g\in G\;:\;g^2\neq a\}$. Here I do not assume
that the group is abelian.
Solution to special subcase 1:
Assume that $G$ is abelian and that $H=\{g\in G\;:\;s(g)\neq t(g)\}$
for some $1$-variable equation $s(x)=t(x)$, possibly with parameters.
Moving the variables to the left of the equals sign and the parameters
to the right, the equation $s(x)=t(x)$ reduces to something of the
form $x^n=a$. Hence the co-equational definition of $H$ is equivalent
to $H=\{g\in G\;:\;x^n\neq a\}$. (Note that $1\in H$, so
$1^n\neq a$, so $a\neq 1$.)
I claim that either (i) $H=G$ and
$H$ is defined equationally with the equation $x=x$
[$H=\{g\in G\;:\;g = g\}$] or
(ii) $H\neq G$ and $H$ is defined equationally
with the equation $x^n=1$
[$H=\{g\in G\;:\;g^n = 1\}$].
To show this, assume that $H\neq G$ and that $p\in G-H$.
By the co-equational definition of $H$, $p^n=a\neq 1$.
Let $W=\{g\in G\;:\;g^n=1\}$.
Now observe that if $h\in H$, then $ph\notin H$, so
$a = (ph)^n = p^nh^n = ah^n$. Canceling $a$ yields $h^n=1$.
This argument shows that $H\subseteq W$.
Conversely, choose any $g\in W$.
Since $g^n=1\neq a$, then $g$ satisfies the co-equational
definition of $H$, so $g\in H$. Altogether this shows that
$H=W$, which means that $H$ is equationally defined by $x^n=1$. $\Box$
Solution to special subcase 2:
I do not assume that I am working with abelian groups,
but I do assume that $H$ is defined co-equationally
by $H=\{g\in G\;:\;g^2\neq a\}$.
Since $1\in H$, we have $1^2\neq a$, so $a\neq 1$.
This implies that $a^2\neq a$, so we have $a\in H$
by the co-equational definition of $H$.
Also, from Kenta S's comment on the original question,
we have $a^2=1$.
Note that this implies that any $p\in G-H$
has order $4$, since $p^4=a^2=1$, but $p^2=a\neq 1$.
I claim that either (i) $H=G$ and
$H$ is defined equationally
with the equation $x=x$,
[$H=\{g\in G\;:\;g = g\}$] or
(ii) $H\neq G$ and $H$ is defined equationally
with the equation $x^2=1$
[$H=\{g\in G\;:\;g^2 = 1\}$], or (iii)
$H\neq G$ and $H$ is defined equationally
with the equation $p^{-1}xp=x^{-1}$ for some
parameter $p\in G-H$
[$H=\{g\in G\;:\;p^{-1}gp = g^{-1}\}$].
Preliminary observation.
If $h\in H$ and $p\in G-H$, then $p^{-1}hp=h^{-1}$. ($p$ conjugates any element of $H$ to its inverse.)
To see this, note that $ph\in G-H$, so $(ph)^2=a=p^2$.
The equation $phph=pp$ may be rewritten as
$(pp)^{-1}phph=1$ and then simplified to $p^{-1}hp=h^{-1}$,
which shows that every element of $H$ satisfies the equation
$p^{-1}xp=x^{-1}$.
Case 1. $[G:H]>2$.
In this case I claim that the equationally-defined subset
$X=\{g\in G\;|\;g^2=1\}$ is exactly $H$.
First, $X\subseteq H$ since
$X$ consists of elements of exponent $2$, while
$G-H$ consists of elements of order $4$.
Conversely, assume that $h\in H$
and that $H, pH, qH$ are distinct cosets of $H$.
Then $p, q, pq\in G-H$ while $h\in H$, so each
of $p, q, pq$ conjugates
$h$ to its inverse. This means that
$h^{-1}=(pq)^{-1}h(pq)=q^{-1}(p^{-1}hp)q=q^{-1}h^{-1}q=h$,
so $h^2=1$ and $h\in X$.
This proves that $H = X$, completing the
proof for Case 1.
Case 2. $[G:H]=2$.
In this case I claim that for any fixed $p\in G-H$
the equationally-defined subset
$Y=\{g\in G\;|\;p^{-1}gp=g^{-1}\}$ is exactly $H$.
We have already shown that $H\subseteq Y$
in the Preliminary Observation, so let's argue
that $Y\subseteq H$. This is equivalent
to showing that $Y\cap (G-H)=\emptyset$.
To obtain a contradiction,
assume that there is some $q\in Y\cap (G-H)$.
Since $q\in Y$ and $p\in G-H$, the Preliminary
Observation guarantees that $p^{-1}qp=q^{-1}$.
Right multiplying by $q$ and left multiplying by $p^2$
yields $pqpq=p^2=a$, so $(pq)^2=a$.
This means that $pq$ fails the co-equational
definition of $H$, so $pq\in G-H$.
But this contradicts the assumption that $[G:H]=2$,
as I now explain.
Under the assumption $[G:H]=2$,
$H\lhd G$ and $G/H$ is a $2$-element group.
For each of $p, q, pq\in H$ we have that
$pH = qH= pqH$ represent the nonidentity
element of this quotient group. By the multiplication
table of the $2$-element group, we should have
$pH\cdot qH = H$, but we have $pH\cdot qH=pqH\neq H$.
This contradiction shows that $Y\cap (G-H)=\emptyset$,
so $Y\subseteq H$, so $Y=H$. This completes
the proof of special subcase 2. $\Box$
Finally, note that if $G=Q_8$ is the $8$-element quaternion
group and $a$ is the nonidentity central element,
then the center
$Z(G) = \{g\in G\;:\;g^2\neq a\}$ is a co-equationally
defined subgroup, and it is equationally
defined by $\{g\in G\;|\;g^2=1\}$ (the Case 1 equational definition).
But $Z(G)$ is NOT equationally
defined by $\{g\in Q_8\;|\;p^{-1}gp=g^{-1}\}$ for any $p\in Q_8$
(the Case 2 equational definition).
On the other hand, if $G=\textrm{Dic}_3$ is the $12$-element dicyclic
group and $a$ is the nonidentity central element,
then the index-$2$ subgroup is defined co-equationally
by $\{g\in G\;:\;g^2\neq a\}$. It is equationally defined by
$\{g\in G\;|\;p^{-1}gp=g^{-1}\}$ (Case 2 equational definition)
using any parameter $p$ of order $4$,
but NOT by $\{g\in G\;|\;g^2=1\}$ (the Case 1 equational definition).
Thus, when $H\neq G$, it seems that you really need to consider two cases.
