Note that $(ab)^2 = b^2a^2$ for given $a$ and $b$ is equivalent to $(ba)^3 = b^3a^3$. Indeed, we have that if $(ab)^2=b^2a^2$, then
$$b^3a^3 = b(b^2a^2)a = b(ab)^2a = bababa = (ba)^3.$$
Conversely, if $(ba)^3 = b^3a^3$, then
$$b^3a^3 = (ba)^3 = b(ab)^2a,$$
and cancelling $b$ on the left and $a$ on the right we get $b^2a^2 = (ab)^2$.
Thus,
$$(ab)^2 = b^2a^2\text{ for all }a,b\in G\iff (xy)^3 = x^3y^3\text{ for all }x,y\in G.$$
Definition. Let $G$ be a group, and $n$ an integer. We say that $G$ is $n$-abelian if and only if $(ab)^n = a^nb^n$ for all $a,b\in G$; equivalently, if and only if the map $x\mapsto x^n$ defines a group endomorphism from $G$ to itself.
It is a standard exercise that a group $G$ is abelian if and only if it is $2$-abelian, if and only if it is $(-1)$-abelian. You are asking whether a $3$-abelian group is necessarily abelian. The answer is "no."
In
Alperin, J.L. A classification of $n$-abelian groups. Canad. J. Math. 21 1969 1238–1244. MR0248204 (40 #1458)
Alperin described the structure of $n$-abelian groups. A group $G$ is $n$-abelian if and only if it is a homomorphic image of a subgroup of a product of the form $A\times M\times N$, where $A$ is abelian, $M$ is a group of exponent $n-1$, and $N$ is a group of exponent $n$.
Since for any $n\gt 2$ there exist nonabelian groups of exponent $n$, it follows that for any $n\gt 2$ there exist $n$-abelian groups that are not abelian in particular, there are $3$-abelian groups that are not abelian. For example, the nonabelian group of order $p^3$ and exponent $p$ for odd prime $p$, which can be realized as the multiplicative group of all unitriangular $3\times 3$ matrices of the form
$$\left(\begin{array}{ccc}
1&a&b\\
0&1&c\\
0&0&1
\end{array}\right),\qquad a,b,c\in\mathbb{Z}/p\mathbb{Z},$$
is nonabelian of exponent $p$, will give examples of $n$-abelian groups any odd $n\gt 2$ (take an odd prime dividing $p$); and for $n=2^a$ with $a\gt 1$, take the dihedral group of order $8$, which has exponent $4$.
Note that a group is $n$-abelian if and only if it is $(1-n)$-abelian: if $(xy)^n = x^ny^n$ for all $x$ and $y$, then
$$(ab)^{1-n} = a(ba)^{-n}b = a\left((a^{-1}b^{-1})^n\right)b = aa^{-n}b^{-n}b = a^{1-n}b^{1-n}.$$
Therefore, if $n\lt 0$, then there exist $n$-abelian groups that are not abelian if and only if $1-n\neq 2$, if and only if $n\neq-1$. So in fact the only $n$s for which $n$-abelian coincides with abelian are $n=-1$ and $n=2$.
An interesting related question is:
For which values of $n,m$ is it the case that if a group $G$ is both $n$-abelian and $m$-abelian, then $G$ is necessarily abelian?
As shown in this math.overflow post, this happens if and only if $\gcd(n^2-n,m^2-n)=2$.
