Since you already have the diagram answer, easily awailable from
wikipedia by typing you question title in Google, as a .png diagram
identical to yours, it should be easy for you to find the formula by
extracting it from that diagram. Given the definition NAND as
$\text{NAND}(A,B)=\overline{AB}\;$:
The leftmost gate gives $C=\overline{AB}$;
The top gate gives $D_1=\overline{AC}$;
The top gate gives $D_2=\overline{BC}$, as the NAND is
commutatve like the AND;
The rightmost gate gives $E=\overline{D_1D_2}$.
Putting it all together we first note that
$C=\overline{AB}=\overline A+\overline B$
$\begin{align}
\overline{D_1}&=AC\\
&=A(\overline A+\overline B)\\
&=A\overline A+A\overline B\\
&=0+A\overline B\\
&=A\overline B\\
\end{align}$
Similarly: $\overline{D_2}=B\overline A$
Thus
$\begin{align}
E&=\overline{D_1D_2}\\
&=\overline{D_1}+\overline{D_2}\\
&=A\overline B + B\overline A
\end{align}$
Which is precisely the definition of XOR. You may just reverse all this if you want to start from your initial data, rather than just check the answer.
Finding the answer with no prior knowledge
This is intended to answer the explicit request, added as an edit to the question, for a way of finding the solution from scratch. Given that the question is about a thought process, I am giving all details.
I would try to rely on the constraints of the problem (only 4 NAND gates) and on its symmetry between $A$ and $B$ which may be preserved in the solution.
One thing I know (assuming information flows from left to right as in the question diagrams) is that there must be a rightmost NAND gate that produces the desired answer $\text{XOR}(A,B)=A\overline B + B\overline A\;$.
So we can try to guess what kind of input to this gate would produce the desired output.
We know that $\text{NAND}(X,Y)=\overline{XY}= \overline X+\overline
Y\;$
Unifying this last formula with the result
we have to get, we obtain:
$\overline X=A\overline B\;$, thus $X=\overline{A\overline B}=\overline A+B\;$.
and symetrically $Y=\overline{\overline A B}=A+\overline B\;$.
Note that this is only the simplest possibility. There are other pairs of inputs that would give the desired result, because we are not unifying in a free algebra, since NAND has equational properties. But we try that for a start.
The problem is now whether we can obtain both $X$ and $Y$ from $A$ and
$B$ with 3 NAND gates.
We could try to repeat the unification procedure (I did), but this will
naturally lead us to using four more gates, hence to a 5 gates solution.
Assuming we are on the right track, we need two NAND gates to produce
$X$ and $Y$. So that leaves us with only one gate to produce a formula
$Z$ that combined with $A$ or $B$ will provide the input for these two
intermediate gates.
Given that we have to provide symetrically for $X$ and $Y$, we can
expect that $Z$ should be symmetric in $A$ and $B$. Hence this
leftmost NAND gate should take both $A$ and $B$ as input.
This first NAND gate, with $A$ and $B$ as input, produces as output:
$Z=\text{NAND}(A,B)=\overline{AB}= \overline A+\overline B\;$
Now, we have to check whether combining $Z$ with itself, $A$, $B$, 0,
or 1 through a NAND gate can produce $X$, and also $Y$.
We know that combining a value with itself, 0 or 1 through a NAND gate
is either the identity function or the negation. So the only remaining
candidates are $A$ and $B$.
It is easy to check that
$\begin{align}
\text{NAND}(Z,A)&=\overline{ZA}\\
&=\overline{\overline{AB}\;A}\\
&=\overline{(\overline A+\overline B)\;A}\\
&=\overline{\overline AA+\overline BA}\\
&=\overline{0+\overline BA}\\
&=\overline{\overline BA}\\
&=\overline{A\overline B}\\
&=X
\end{align}$
Similarly $\text{NAND}(Z,B)=Y$
Hence we can compose these four gates to get the desired result, i.e.,
the XOR function.
