Consider elements $a$, $b$ in G
$(ab)^{2} = e$ --- We are told this is true.
$(ab)(ab) = e$ --- expanding the square.
$ abab = e$ --- Associativity (remember we are in a group, the binary operation is associative) means brackets don't really matter here.
Now if we multiply both sides on the left by $a$ and the right by $b$ we obtain
$a(abab)b = eab$
However $aa = a^{2} = e$ and $bb = b^2 = e$. Also, $eab = ab$ Therefore we see that
$ba = ab$ for all elements $a$ and $b$ in $G$ --- That is to say, $G$ is Abelian.
I would like to make a note on your question "what is the operation?" --- when you are thinking about a an abstract group $G$ it doesn't matter what the operation is exactly. However, it is helpful to treat it like addition sometimes, or multiplication other times (in this case it was treated like multiplication). But know that you're not really doing either, you're applying a general binary map with the properties that make it feel like multiplication or addition (associativity, inverses and identities)