This is not as hard as it seems.
[I will slightly alter your notation, and instead of $(m+n)^2$ I will use $(m + n\sqrt{c})^2$.]
$$ (m + n\sqrt{c})^2 = a + b \sqrt{c} $$
Expanding and collecting everything on the left-hand side gives us:
$$ n^2 c + (2mn - b) \sqrt{c} + (m^2 - a) = 0$$
We are given $a, b, c \in \mathbb{Z}$, and likewise we want $m, n \in \mathbb{Z}$.
Well, the first and third terms are integers, and the second one isn't (unless $c$ is a perfect square -- otherwise we could have used simpler methods). This means that the coefficient of $\sqrt{c}$ must be equal to $0$. This gives us the new equation $mn = \frac{b}{2}$. So, $b$ must be even for this to work.
The two remaining terms also need to sum to 0. This means $n^2 c + m^2 = a$.
So, let's sum up what we have so far:
- $b$ must be even.
- $mn = \frac{b}{2}$
- $n^2 c + m^2 = a$
Unless $b = 0$, in which case we have a much simpler problem, we have that $n \neq 0$. This means that we can safely divide by $n$ , and get:
We can substitute that into our last equation, which gives us:
- $n^2 c + \frac{b^2}{4n^2} = a$
If we let $N = n^2$, we can multiply by $N$ and rewrite all this as:
- $N^2 c + \frac{b^2}{4} = a$
So, we get: $N^2 = \frac{a}{c} - \frac{b^2}{4c}$.
This means:
$$N^2 = \frac{4a - b^2}{4c}$$
The right-hand side must be positive (so $4a > b^2$) and it must be a perfect square.
So, the constraint you were looking for are:
- $b$ is even.
- $4a > b^2$
- $\frac{4a - b^2}{4c}$ is a perfect square.
[The first constraint is not strictly necessary, if you don't mind working with fractions instead of integers.]
For your example, we get $$\frac{4\times 3 - 2^2}{4 \times 2} = \frac{8}{8} = 1 = 1^2$$
So, we know $n = \pm \sqrt{N}$, and $m = \pm \frac{b}{2\sqrt{N}}$. We can safely choose one sign for $n$ and take the corresponding sign for $m$ (the other option would just give us $(-m -n\sqrt{c})$).