There are a number of things that need to be calculated before arriving at the final answer.
First we keep in mind that $P(B_{\tau}=a)=\frac{b}{b-a}$ and $P(B_{\tau}=b)=\frac{a}{a-b}$.
Consider the martingale $B_t^2-t$.
$$
\begin{align*}
E(B_{\tau}^2-\tau)&=0 \\
E(B_{\tau}^2\mid B_{\tau}=a)P(B_{\tau}=a)+E(B_{\tau}^2\mid B_{\tau}=b)P(B_{\tau}=b)-E(\tau)&=0 \\
a^2\frac{b}{b-a}+b^2\frac{a}{a-b}-E(\tau)&=0 \\
E(\tau)&=\frac{a^2b-b^2a}{b-a} \\
E(\tau)&=-ab.
\end{align*}
$$
Use the first hint in the link to get an equation involving $E(\tau\mid B_{\tau}=a)$ and $E(\tau\mid B_{\tau}=b)$.
$$
\begin{align*}
E(B_{\tau}^3-3\tau B_{\tau})&=0 \\
E(B_{\tau}^3\mid B_{\tau}=a)P(B_{\tau}=a)+E(B_{\tau}^3\mid B_{\tau}=b)P(B_{\tau}=b) \\
-3E(\tau B_{\tau}\mid B_{\tau}=a)P(B_{\tau}=a)-3E(\tau B_{\tau}\mid B_{\tau}=b)P(B_{\tau}=b)&=0 \\
a^3\frac{b}{b-a}+b^3\frac{a}{a-b}-3aE(\tau\mid B_{\tau}=a)\frac{b}{b-a}-3bE(\tau\mid B_{\tau}=b)\frac{a}{a-b}&=0 \\
a^3b-b^3a-3abE(\tau\mid B_{\tau}=a)+3abE(\tau\mid B_{\tau}=b)&=0 \\
E(\tau\mid B_{\tau}=b)-E(\tau\mid B_{\tau}=a)&=\frac{b^2-a^2}{3}.
\end{align*}
$$
We have
$$
\begin{align*}
E(\tau\mid B_{\tau}=a)P(B_{\tau}=a)+E(\tau\mid B_{\tau}=b)P(B_{\tau}=b)&=E(\tau) \\
E(\tau\mid B_{\tau}=a)\frac{b}{b-a}+E(\tau\mid B_{\tau}=b)\frac{a}{a-b}&=-ab \\
bE(\tau\mid B_{\tau}=a)-aE(\tau\mid B_{\tau}=b)&=-ab(b-a).
\end{align*}
$$
Combine this with the equation earlier to get a system of equations you need to solve.
You will find that $E(\tau\mid B_{\tau}=a)=\frac{a^2-2ab}{3}$ and $E(\tau\mid B_{\tau}=b)=\frac{b^2-2ab}{3}$. Now use the second hint in the link.
$$ \begin{align} E(B_{\tau}^4-6\tau B_{\tau}^2+3\tau^2)&=0 \ E(B_{\tau}^4\mid B_{\tau}=a)P(B_{\tau}=a)+E(B_{\tau}^4\mid B_{\tau}=b)P(B_{\tau}=b) \ -6E(\tau B_{\tau}^2\mid B_{\tau}=a)P(B_{\tau}=a)-6E(\tau B_{\tau}^2\mid B_{\tau}=b)P(B_{\tau}=b) \ +3E(\tau^2)&=0 \ a^4\frac{b}{b-a}+b^4\frac{a}{a-b}-6a^2\frac{b}{b-a}E(\tau\mid B_{\tau}=a)-6b^2\frac{a}{a-b}E(\tau\mid B_{\tau}=b)+3E(\tau^2)&=0 \ a^4\frac{b}{b-a}+b^4\frac{a}{a-b}-6a^2\frac{b}{b-a}\frac{a^2-2ab}{3}-6b^2\frac{a}{a-b}\frac{b^2-2ab}{3}+3E(\tau^2)&=0 \ ab(a^2-3ab+b^2)+E(\tau^2)&=0. \end{align} $$
Therefore $E(\tau^2)=-ab(a^2-3ab+b^2)$.