An answer that does not use hypergeometric functions:
First integrate by parts using the functions $u(x)=x\ln x$ and $v(x)=-\sqrt{1-x^2}$, we have $u'(x)=\ln x +1$ and $v'(x)=\frac{x}{\sqrt{1-x^2}}$ and we get
$$I=\int_0^1\frac{x^2\ln x}{\sqrt{1-x^2}}\mathrm dx=
\underbrace{\left[-x\ln x\sqrt{1-x^2}\right]_0^1}_{=0}+\int_0^1(\ln x+1)\sqrt{1-x^2}\,\mathrm dx.$$
The integral $\int_0^1\sqrt{1-x^2}\mathrm dx=\frac\pi4$ is easy. Let us concentrate on
$$J=\int_0^1\ln x \,\sqrt{1-x^2}\mathrm dx$$
for which it seems a trigonometric change of variable will work. Let us set $x=\cos\theta$, $\mathrm dx=-\sin\theta\mathrm d\theta$, then
$$J=\int_0^{\pi/2}\ln(\cos\theta)\sin^2\theta\mathrm d\theta
=\int_0^{\pi/2}\ln(\cos\theta)(1-\cos^2\theta)\mathrm d\theta.\tag1$$
We can use the result
$$\int_0^1\ln(\sin\theta)\,\mathrm d\theta=\int_0^{\pi/2}\ln(\cos\theta)\,\mathrm d\theta=-\frac\pi2\ln2$$
(see for instance this post for a derivation). Therefore we have
$$J=-\frac\pi2\ln2-\int_0^{\pi/2}\ln(\cos\theta)\cos^2\theta\,\mathrm d\theta.\tag2$$
Adding up (1) and (2) we obtain
$$2J=-\frac\pi2\ln2+\int_0^{\pi/2}\ln(\cos\theta)\left(\sin^2\theta-\cos^2\theta\right)\,\mathrm d\theta=-\frac\pi2\ln2-\int_0^{\pi/2}\ln(\cos\theta)\cos(2\theta)\,\mathrm d\theta.$$
Finally let us integrate by parts (with $u(x)=\ln(\cos\theta)$ and $v(x)=\frac12\sin(2\theta)$, $u'(x)=-\tan\theta$ and $v'(x)=\cos(2\theta)$)
$$\begin{split}\int_0^{\pi/2}\ln(\cos\theta)\cos(2\theta)\,\mathrm d\theta&=
\underbrace{\left[\ln(\cos\theta)\frac12\sin(2\theta)\right]_0^{\pi/2}}_{=0}
+\int_0^{\pi/2}\tan\theta\,\frac12\sin2\theta\mathrm d\theta\\
&=\int_0^{\pi/2}\sin^2\theta\,\mathrm d\theta=\frac\pi4
\end{split}$$
We get $2J=-\frac\pi2\ln2-\frac\pi4$ and therefore
$$I=-\frac\pi4\ln2-\frac\pi8+\frac\pi4=\boxed{\frac\pi8-\frac\pi4\ln2}.$$
$$\int_0^1\frac{x^{2n}\ln x}{\sqrt{1-x^2}}dx =I_n=\frac{2n-1}{2n}I_{n-1}+\frac1{2n}\int_0^1 x^{2(n-1)}\sqrt{1-x^2}>dx $$
– Quanto Apr 05 '22 at 13:15