As pointed in the comments, I overlooked some details in my previous answer. I am editing it to reflect on some relevant points.
We already know an upper bound for the expression is $4$.
In short, you can represent these inequalities as parabolas in the $1$st quadrant.
$$b^2-4a \le 0$$
$$a^2-4a \le 0$$
$$(a-2)^2+(b-2)^2 = (2\sqrt{2})^2$$
You can interpret the first two inequalities as parabolas and the third as a circle.

To find a maximum, you need to consider points that lie in the region that satisfies the first two inequalities (which is just the region inside the two parabolas), and are closest to the circle described by the third equation.
It is easy to see that in this case the upper bound $4$ is achieved with $(4,4)$.
But if you want distinct equations, you can try to find points with distinct values for the $x$ and $y$ coordinate. Clearly, there is no maximum in this case, but only a strict upper bound of $4$.
If you consider the problem over the distinct integers, this condition is achieved with $(2,1)$ only.