Let us write $x^2y^2 -2(x+y) = (xy-a)^2$. Then, $a(2xy-a) = 2(x+y)$. This means that $a$ must be even, because $2(x+y)$ is even, and $a$ and $2xy-a$ have the same parity.
To give you an intuition about our idea, let us think about the following question: What is the difference of squares of consecutive numbers, or consecutive even numbers, or numbers of an arithmetic progression, as they get larger? The answer is quite simple: As the numbers get larger, the differences get larger.
Now, suppose you have a very large number, and you are to subtract something from it to get a square, as is happening here (with $x^2y^2$ being a very large number, $2(x+y)$ being the subtrahend, and $(xy-a)^2$ being the resulting square). Whatever is being subtracted needs to be large enough to be able to get to the square nearest to the large number, isn't it? This idea will be key to our proof.
Now, think about this: from a very large number, $x^2y^2$, we are subtracting a very small quantity, $2(x+y)$. Will this quantity be enough to reach the previous square? Note that above I just proved that $a$ has to be even, hence the largest square smaller than $x^2y^2$ which we can possibly reach by subtracting $2(x+y)$ is $(xy-2)^2$. All other squares which can possibly be reached by subtracting $2(x+y)$ all have to be smaller than $(xy-2)^2$. Hence, it is wise to check the case $a=2$. That is, we are asking the question: For what $x$ and $y$ is $2(x+y)$ so large enough that subtracting it from $x^2y^2$ can lead to $(xy-2)^2$?
For $a=2$, the difference we are looking to overcome is $a(2xy-a) = 2(2xy-2) = 4xy-4$. We are hence supposed to find all $x$ and $y$ such that $2(x+y)$ is larger than $4xy-4$, for then it will be large enough to result in a perfect square when subtracted, whether that perfect square be $(xy-2)^2$ or $(xy-4)^2$ or $(xy-72)^2$. We are just looking for whether it is large enough to cross the threshold or not.
Now, a chain of equalities: $2(x+y) \geq 4xy-4 \iff 2y+4 \geq x(4y-2) \iff x \leq \frac{2y+4}{4y-2} =\frac{y+2}{2y-1} =\frac{1}{2}+\frac{5}{2(2y-1)}$.
Now, we can verify that $x \leq 3$, because the maximum of the right hand side is $3$, achieved when $y=1$.
Thus, forget about large values of $a$, for $a=2$ itself, only small values $(x \leq 3)$ are guaranteed to give you a large enough value of $2(x+y)$ such that upon subtraction, it is able to reach the previous square. Which is why we do not need to go for general value of $a$ : if $2(x+y)$ was large enough to reach $(xy-a)^2$, then surely it was also large enough to reach $(xy-2)^2$, wasn't it? But then, that again could happen only when $x=3$! That's why I don't bother about general $a$, and start examining values of $x$!
Now we can individually verify the solutions:
Take $x=1$. Then $x^2y^2-2(x+y) = y^2-2y-2 = (y-1)^2 - 3$. This a square only when the difference between two squares is three, which happens only with $1^2$ and $2^2$. Hence, $y-1=2$ and $y=3$.
Take $x=2$. Then $x^2y^2-2(x+y) = 4y^2-2y-4$. Now, I claim that $4y^2-2y-4$ can never be the square of a positive number. For if $4y^2-2y-4=z^2$, then:
$$
y=\frac{1+\sqrt{(2z)^2+17}}{4}
$$
The only two squares differing by $17$ are $9^2=81$ and $8^2=64$. But then $y=\frac{10}{4}$ in this case, hence we do not get an integer value of $y$. Hence there is no solution for $x=2$.
Take $x=3$.Then $x^2y^2-2(x+y) = 9y^2-2y-6 $. Now, I claim that $9y^2-2y-6$ can never be the square of a positive number, except for $y=1$. For if $9y^2-2y-6=z^2$, then:
$$
y=\frac{1+\sqrt{(3z)^2+55}}{9}
$$
All pairs of squares differing by $55$ are the following: $8^2-3^2=55,28^2-27^2=55$. The first case gives $y=\frac{1+8}{9} =1$ and the second case gives $y = \frac{1+28}{9} = \frac{29}{9}$, not an integer. Hence $y=1$ is the only value.
Hence we conclude that $x=1,y=3$ and $x=3,y=1$ are the only permissible values.
