Quanto's derivation above is valid for $|a|<1$.
Now, let us consider the case $|a|>1$. Let $b = 1/a$, so $|b|<1$.
The argument of the logarithm in the integral can be rewritten as:
$$ \frac{1+a^2+2a\cos(x)}{1+a^2-2a\cos(x)} = \frac{a^2(1/a^2+1+2/a\cos x)}{a^2(1/a^2+1-2/a\cos x)} = \frac{1+b^2+2b\cos(x)}{1+b^2-2b\cos(x)} $$
The expression inside the integral is unchanged if we replace $a$ with $1/a$. Let $I(a)$ be the value of the integral for a given $a$. Then $I(a) = I(1/a)$.
So, for $|a|>1$, we can use the result for $|b|<1$:
$$ I(a) = I(b) = 4\pi(\text{Li}_2(b^2) - \text{Li}_2(-b^2)) = 4\pi\left(\text{Li}_2\left(\frac{1}{a^2}\right) - \text{Li}_2\left(-\frac{1}{a^2}\right)\right) $$
For $|a|=1$, the integrand has a singularity at one of the endpoints of the interval. At this point, the value of the integrand diverges to infinity. However, this is an integrable singularity.
Credit to Gary, we have (for $a=1, -1$): $$4 \int_{0}^{\pi} \log^2 \left( \tan \left( \frac{x}{2} \right) \right) \, dx = 8 \int_{0}^{+\infty} \frac{\log^2t}{1 + t^2} \, dt = \pi^3$$
Thus, the value of the integral is given by:
$$ \int^{\pi}_{0}\ln^2\left({\frac{1+a^2+2a\cos(x)}{1+a^2-2a\cos(x)}}\right) \,dx = \begin{cases} 4\pi(\text{Li}_2(a^2) - \text{Li}_2(-a^2)) & \text{if } |a| < 1 \\
\pi^3 & \text{if } |a| = 1 \\
4\pi\left(\text{Li}_2\left(\frac{1}{a^2}\right) - \text{Li}_2\left(-\frac{1}{a^2}\right)\right) & \text{if } |a| > 1 \end{cases} $$
Numerical Verification using scipy
| $a$ |
Numerical Result |
Analytical Result |
| $0.5$ |
$2.505504791515$ |
$2.505504791515$ |
| $0.8$ |
$13.921356064019$ |
$13.921356064019$ |
| $2.0$ |
$2.505504791515$ |
$2.505504791515$ |
| $5.0$ |
$0.495753852554$ |
$0.495753852554$ |
Edit
Interestingly, we can further write the solutions to the integral in terms of the Legendre Chi Function of order 2.
The definition of $\chi_2(z)$ is:
$$ \chi_2(z) = \sum_{k=0}^{\infty} \frac{z^{2k+1}}{(2k+1)^2} = z + \frac{z^3}{9} + \frac{z^5}{25} + \cdots \quad (\text{for } |z| \le 1) $$
Using this we can say that:
$$ \boxed{\text{Li}_2(z) - \text{Li}_2(-z) = 2 \chi_2(z)} $$
This comes directly from the series definitions:
$$\text{Li}_2(z) = \sum_{n=1}^{\infty} \frac{z^n}{n^2} = \frac{z}{1^2} + \frac{z^2}{2^2} + \frac{z^3}{3^2} + \frac{z^4}{4^2} + \cdots$$
$$\text{Li}_2(-z) = \sum_{n=1}^{\infty} \frac{(-z)^n}{n^2} = -\frac{z}{1^2} + \frac{z^2}{2^2} - \frac{z^3}{3^2} + \frac{z^4}{4^2} - \cdots$$
Subtracting the second series from the first, all the even-power terms cancel out, and the odd-power terms double:
$$ \text{Li}_2(z) - \text{Li}_2(-z) = 2\left(\frac{z}{1^2} + \frac{z^3}{3^2} + \frac{z^5}{5^2} + \cdots\right) = 2\chi_2(z) $$
Using this identity, we can rewrite the result much more cleanly.
For $|a| < 1$ with $z=a^2$,
$$ 4\pi \times (2\chi_2(a^2)) = 8\pi \chi_2(a^2) $$
For $|a| > 1$ with $z=1/a^2$,
$$ 4\pi \times \left(2\chi_2\left(\frac{1}{a^2}\right)\right) = 8\pi \chi_2\left(\frac{1}{a^2}\right) $$
Thus, the value of the integral can be written more elegantly as:
$$ \int^{\pi}_{0}\ln^2\left({\frac{1+a^2+2a\cos(x)}{1+a^2-2a\cos(x)}}\right) \,dx = \begin{cases} 8\pi \chi_2(a^2) & \text{if } |a| < 1 \\
\pi^3 & \text{if } |a| = 1 \\
8\pi \chi_2\left(\frac{1}{a^2}\right) & \text{if } |a| > 1 \end{cases} $$