I want to evaluate the integral $\int_0^{\frac{\pi}{2}} \ln|\sin^2x-a^2|dx$ where $a^2\leqslant 1$.
And I have the Euler’s integral $\int_0^{\frac{\pi}{2}} \ln\sin x dx=-\frac{\pi}{2}\ln{2}$, but it seems that I have difficulties in substitution.
Let $$ I(t)=\int_0^{\frac{\pi}{2}} \ln|t-\sin^2x|dx, t\ge1. $$ Then \begin{eqnarray} I'(t)&=&\int_0^{\frac{\pi}{2}} \frac{1}{t-\sin^2x}dx\\ &=&\int_0^{\frac{\pi}{2}} \frac{\csc^2x}{t\csc^2x-1}dx\\ &=&-\int_0^{\frac{\pi}{2}} \frac{1}{t(\cot^2x+1)-1}d\cot x\\ &=&-\int_{\infty}^{0} \frac{1}{t(u^2+1)-1}du\\ &=&\frac1t\int^{\infty}_{0} \frac{1}{u^2+\frac{t-1}t}du\\ &=&\frac1t\sqrt{\frac t{t-1}}\arctan(u\sqrt{\frac t{t-1}})\bigg|^{\infty}_{0}\\ &=&\frac{\pi}{2t}\sqrt{\frac t{t-1}}. \end{eqnarray} Now it is easy to obtain (I omit the detail) $$ I(t)-I(1)=\int_1^t\frac{\pi}{2t}\sqrt{\frac t{t-1}}dt=\text{arcsinh}(\sqrt{t-1}) $$ and hence $$ I(a^2)=I(1)+\text{arcsinh}(a^2-1)=-\pi\ln2+\text{arcsinh}(\sqrt{a^2-1}). $$
You may just use Riemann sums.
For any $n\in\mathbb{N}^+$:
$$ x^{2n}-1 = \prod_{k=1}^{2n}\left(x-\exp\frac{\pi i k}{n}\right) = (x^2-1)\prod_{k=1}^{n-1}\left(x^2+1-2x\cos\frac{\pi k}{n}\right).$$
If we choose $x\in\mathbb{C}$ in such a way that $\frac{x^2+1}{2x}=a$ holds, for instance through $x=a+i\sqrt{1-a^2}$,
we get:
$$ \int_{0}^{\pi}\log(a-\cos\theta)\,d\theta = \lim_{n\to +\infty} \frac{\pi}{n}\log\prod_{k=1}^{n-1}\left(a-\cos\tfrac{\pi k}{n}\right) = \pi \log\left(\frac{a+i\sqrt{1-a^2}}{2}\right).$$
We may write the LHS as $\int_{0}^{\pi/2}\log(a-\cos\theta)+\log(a+\cos\theta)\,d\theta $ and the RHS as $-\pi\log 2 + i\pi\arccos(a)$. With some care in managing the determinations of the complex logarithm,
$$ \int_{0}^{\pi/2}\log\left|a^2-\cos^2\theta\right|\,d\theta = \int_{0}^{\pi/2}\log\left|a^2-\sin^2\theta\right|\,d\theta = \color{blue}{-\pi\log 2} $$
for any $a\in(-1,1)$ is a straightforward consequence. This can be seen as a consequence of the properties of the Poisson kernel, too: have a look at page 55 of my notes.
For $a^2\le1$, let $a=\sin t$ where $t\in[0,\pi/2]$. Then \begin{eqnarray} I&=&\int_0^{\frac{\pi}{2}} \ln|\sin^2t-\sin^2x|dx\\ &=&\int_0^{\frac{\pi}{2}} \ln|(\sin t-\sin x)(\sin t-\sin x)|dx\\ &=&\int_0^{\frac{\pi}{2}} \ln|\sin(x-t)\sin(x+t)|dx\\ &=&\int_0^{\frac{\pi}{2}} \ln|\sin(x-t)|dx+\int_0^{\frac{\pi}{2}} \ln|\sin(x+t)|dx\\ &=&\int_{-t}^{\frac{\pi}{2}-t} \ln|\sin(x)|dx+\int_{t}^{\frac{\pi}{2}+t} \ln|\sin(x)|dx. \end{eqnarray} Let $$ I(t)=\int_{-t}^{\frac{\pi}{2}-t} \ln|\sin(x)|dx+\int_{t}^{\frac{\pi}{2}+t} \ln|\sin(x)|dx. $$ Clearly $$ I'(t)=-\ln|\sin(\frac{\pi}{2}-t)|+\ln|\sin(-t)|+\ln|\sin(\frac{\pi}{2}+t)|-\ln|\sin(t)|=0 $$ and hence $$ I(t)=I(0)=-\pi\ln 2.$$