I am trying to find closed form for this integral:
$$I(a)=\int_0^1\frac{\arctan{ax}}{\sqrt{1-x^{2}}}dx$$ where$a>0$.
My try: Let: $$I(a)=\int_0^1\frac{\arctan{ax}}{\sqrt{1-x^{2}}}dx$$
Then: $$\frac{dI(a)}{da}=\int_0^1\frac{x}{(1+(ax)^2)\sqrt {1-x^2}}dx$$
How I can complete this work, or is there another way to approach it?
The integral can be expressed in terms of Legendre chi function: $$I(a)=\int_0^1\frac{\arctan\left(ax\right)}{\sqrt{1-x^{2}}}dx\overset{x=\sin \theta}=\int_0^\frac{\pi}{2}\arctan(a\sin \theta)d\theta =2\chi_2\left(\frac{\sqrt{1+a^2}-1}{a}\right)$$
For the first case you asked (before the edit) we can take a special value from here (see $8$): $$\chi_2\left({\sqrt{5}-2}\right)=\frac{\pi^2}{24}-\frac34 \ln^2 (\phi), \quad \phi=\frac{1+\sqrt 5}{2}$$
So for $a=\frac12$ we have: $$I=\int_0^1 \frac{\arctan\left(\frac{x}{2}\right)}{\sqrt{1-x^2}}dx=2\chi_2\left({\sqrt{5}-2}\right)=\frac{\pi^2}{12}-\frac32 \ln^2 (\phi)$$
A series representation for the integral.
For $|x|\leq1$ we have that $$\arctan x=\sum_{n\geq0}\frac{(-1)^n}{2n+1}x^{2n+1}$$ So Your integral is $$I(a)=\int_0^1\sum_{n\geq0}\frac{(-1)^n a^{2n+1}}{2n+1}\frac{x^{2n+1}dx}{\sqrt{1-x^2}}\\ =\sum_{n\geq0}\frac{(-1)^n a^{2n+1}}{2n+1}\int_0^1\frac{x^{2n+1}dx}{\sqrt{1-x^2}}$$ Then we focus on $$\int_0^1\frac{x^{2n+1}dx}{\sqrt{1-x^2}}\overset{x=\sin(t)}=\int_0^{\pi/2}\sin(t)^{2n+1}dt=\frac{2n+1}{2^n}\cdot\frac{(2n)!}{n!}$$ Which is similar to the integral here. So we have $$I(a)=\sum_{n\geq0}\frac{(-1)^n(2n)!}{2^n n!}a^{2n+1}=a\sum_{n\geq0}\frac{(2n)!}{n!}\left(-\frac{a^2}{2}\right)^n$$ Then we set $$t_n=\frac{(2n)!}{n!}\left(-\frac{a^2}{2}\right)^n$$ We have that $$\frac{t_{n+1}}{t_n}=(n+1)(n+1/2)\frac{-2a^2}{n+1}$$ and $t_0=1$. Hence we have the hypergeometric series representation $$I(a)=a\cdot{{}_{2}F_{0}}\left({1,\frac12 \atop -}\bigg|-2a^2\right)$$
Assuming $a$ to be real and positive, a CAS gives
$$\color{blue}{I(a)=\frac{\pi ^2}{4}-2 \sinh ^{-1}(a) \,\coth ^{-1}\left(a+\sqrt{a^2+1}\right)+\text{Li}_2\left(a-\sqrt{a^2+1}\right)-\text{Li}_2\left(\frac{1}{a+\sqrt{a^2+1}} \right)}$$ $$I\left(\frac{1}{2}\right)=\frac{\pi ^2}{12}-\frac{1}{2} \sinh ^{-1}(2) \,\text{csch}^{-1}(2)$$
Using @clathratus's approach $$I(a)=\sum_{n\geq0}\frac{(-1)^n a^{2n+1}}{2n+1}\int_0^1\frac{x^{2n+1}}{\sqrt{1-x^2}}dx$$ we have $$\int_0^1\frac{x^{2n+1}}{\sqrt{1-x^2}}dx=\frac{\sqrt{\pi }\, \Gamma (n+1)}{2\, \Gamma \left(n+\frac{3}{2}\right)}$$ making $$I(a)=a \, _3F_2\left(\frac{1}{2},1,1;\frac{3}{2},\frac{3}{2};-a^2\right)$$ the simplication of which giving the first result.
\begin{align} &\int_0^1\frac{\arctan{ax}}{\sqrt{1-x^{2}}}dx = \int_0^1\int_0^a \frac x{\sqrt{1-x^2}(1+y^2 x^2)}dy \ dx\\ &=\int_0^a \frac {\sinh^{-1}y}{y \sqrt{1+y^2}} {dy}= \int^{\frac{a}{\sqrt{1+a^2}+1}}_0\frac{\ln \frac{1+t}{1-t}}{t}dt \>\>\>\>\>\small{({\sinh^{-1}y=\ln\frac{1+t}{1-t}} )}\\ &= \text{Li}_2\bigg(\frac{a }{\sqrt{1+a^2}+1} \bigg) - \text{Li}_2\bigg(-\frac{a }{\sqrt{1+a^2}+1} \bigg) \end{align}
I would like to consider a generalization of this problem.
Define the function $\mathcal{I}:\mathbb{R}\times[-1,1]\rightarrow\mathbb{R}$ via the definite integral
$$\mathcal{I}{\left(a,b\right)}:=\int_{0}^{1}\mathrm{d}x\,\frac{b\arctan{\left(ax\right)}}{\sqrt{1-b^{2}x^{2}}},$$
where the real arctangent may be given by the usual integral representation
$$\arctan{\left(z\right)}=\int_{0}^{z}\mathrm{d}y\,\frac{1}{1+y^{2}};~~~\small{z\in\mathbb{R}}.$$
Suppose $0<a\land0<b\le1$, and set $p:=\frac{a}{b}>0\land\beta:=\arcsin{\left(b\right)}\in\left(0,\frac{\pi}{2}\right]$.
Next, set $z:=\tan{\left(\frac{\beta}{2}\right)}\in(0,1]\land c:=p+\sqrt{1+p^{2}}=\frac{1}{\sqrt{1+p^{2}}-p}>1$.
$$\begin{align} \mathcal{I}{\left(a,b\right)} &=\int_{0}^{1}\mathrm{d}x\,\frac{b\arctan{\left(ax\right)}}{\sqrt{1-b^{2}x^{2}}}\\ &=\int_{0}^{b}\mathrm{d}y\,\frac{\arctan{\left(\frac{a}{b}y\right)}}{\sqrt{1-y^{2}}};~~~\small{\left[x=b^{-1}y\right]}\\ &=\int_{0}^{\arcsin{\left(b\right)}}\mathrm{d}\varphi\,\arctan{\left(\frac{a}{b}\sin{\left(\varphi\right)}\right)};~~~\small{\left[y=\sin{\left(\varphi\right)}\right]}\\ &=\int_{0}^{\beta}\mathrm{d}\varphi\,\arctan{\left(p\sin{\left(\varphi\right)}\right)}\\ &=\int_{0}^{\tan{\left(\frac{\beta}{2}\right)}}\mathrm{d}t\,\frac{2}{1+t^{2}}\arctan{\left(p\sin{\left(2\arctan{\left(t\right)}\right)}\right)};~~~\small{\left[\varphi=2\arctan{\left(t\right)}\right]}\\ &=\int_{0}^{\tan{\left(\frac{\beta}{2}\right)}}\mathrm{d}t\,\frac{2}{1+t^{2}}\arctan{\left(\frac{2pt}{1+t^{2}}\right)}\\ &=\int_{0}^{\tan{\left(\frac{\beta}{2}\right)}}\mathrm{d}t\,\frac{2}{1+t^{2}}\left[\arctan{\left(\frac{t}{\sqrt{1+p^{2}}-p}\right)}-\arctan{\left(\frac{t}{\sqrt{1+p^{2}}+p}\right)}\right]\\ &=\int_{0}^{z}\mathrm{d}t\,\frac{2}{1+t^{2}}\left[\arctan{\left(ct\right)}-\arctan{\left(c^{-1}t\right)}\right]\\ &=\int_{0}^{z}\mathrm{d}t\,\frac{2\arctan{\left(ct\right)}}{1+t^{2}}-\int_{0}^{z}\mathrm{d}t\,\frac{2\arctan{\left(c^{-1}t\right)}}{1+t^{2}}\\ &=\mathcal{J}{\left(c,z\right)}-\mathcal{J}{\left(c^{-1},z\right)},\\ \end{align}$$
where in the last line above we've introduced the auxiliary function $\mathcal{J}:\mathbb{R}^{2}\rightarrow\mathbb{R}$, defined by
$$\mathcal{J}{\left(c,z\right)}:=\int_{0}^{z}\mathrm{d}t\,\frac{2\arctan{\left(ct\right)}}{1+t^{2}}.$$
Suppose $0<c\land0<z$. Setting $\omega:=2\arctan{\left(z\right)}\in(0,\pi)$, we have $z=\tan{\left(\frac{\omega}{2}\right)}$ and $\frac{1-z^{2}}{1+z^{2}}=\cos{\left(\omega\right)}$.
$$\begin{align} \mathcal{J}{\left(c,z\right)} &=\int_{0}^{z}\mathrm{d}t\,\frac{2\arctan{\left(ct\right)}}{1+t^{2}}\\ &=\int_{0}^{z}\mathrm{d}t\,\frac{2}{1+t^{2}}\int_{0}^{1}\mathrm{d}x\,\frac{ct}{1+c^{2}x^{2}t^{2}}\\ &=\int_{0}^{z}\mathrm{d}t\int_{0}^{1}\mathrm{d}x\,\frac{2ct}{(1+t^{2})(1+c^{2}x^{2}t^{2})}\\ &=\int_{0}^{1}\mathrm{d}x\int_{0}^{z}\mathrm{d}t\,\frac{2ct}{(1+t^{2})(1+c^{2}x^{2}t^{2})}\\ &=\int_{0}^{1}\mathrm{d}x\int_{0}^{z^{2}}\mathrm{d}u\,\frac{c}{(1+u)(1+c^{2}x^{2}u)};~~~\small{\left[t^{2}=u\right]}\\ &=\int_{0}^{1}\mathrm{d}x\int_{0}^{z^{2}}\mathrm{d}u\,\frac{c}{1-c^{2}x^{2}}\left[\frac{1}{1+u}-\frac{c^{2}x^{2}}{1+c^{2}x^{2}u}\right]\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{c}{1-c^{2}x^{2}}\left[\int_{0}^{z^{2}}\mathrm{d}u\,\frac{1}{1+u}-\int_{0}^{z^{2}}\mathrm{d}u\,\frac{c^{2}x^{2}}{1+c^{2}x^{2}u}\right]\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{c}{1-c^{2}x^{2}}\left[\ln{\left(1+z^{2}\right)}-\ln{\left(1+c^{2}x^{2}z^{2}\right)}\right]\\ &=\int_{0}^{c}\mathrm{d}y\,\frac{1}{1-y^{2}}\left[\ln{\left(1+z^{2}\right)}-\ln{\left(1+y^{2}z^{2}\right)}\right];~~~\small{\left[cx=y\right]}\\ &=-\int_{0}^{c}\mathrm{d}y\,\frac{1}{1-y^{2}}\ln{\left(\frac{1+z^{2}y^{2}}{1+z^{2}}\right)}\\ &=-\int_{1}^{\frac{1-c}{1+c}}\mathrm{d}t\,\frac{(-1)}{2t}\ln{\left(\frac{1+z^{2}(\frac{1-t}{1+t})^{2}}{1+z^{2}}\right)};~~~\small{\left[y=\frac{1-t}{1+t}\right]}\\ &=-\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\frac{1}{2t}\ln{\left(\frac{(1+t)^{2}+z^{2}(1-t)^{2}}{(1+z^{2})(1+t)^{2}}\right)}\\ &=-\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\frac{1}{2t}\ln{\left(\frac{(1+z^{2})+2(1-z^{2})t+(1+z^{2})t^{2}}{(1+z^{2})(1+t)^{2}}\right)}\\ &=-\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\frac{1}{2t}\ln{\left(\frac{1+2t(\frac{1-z^{2}}{1+z^{2}})+t^{2}}{(1+t)^{2}}\right)}\\ &=-\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\frac{1}{2t}\ln{\left(\frac{1+2t\cos{\left(\omega\right)}+t^{2}}{(1+t)^{2}}\right)}\\ &=\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\frac{1}{2t}\left[2\ln{\left(1+t\right)}-\ln{\left(1+2t\cos{\left(\omega\right)}+t^{2}\right)}\right]\\ &=\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\left[\frac{\ln{\left(1+t\right)}}{t}-\frac{\ln{\left(1+2t\cos{\left(\omega\right)}+t^{2}\right)}}{2t}\right]\\ &=\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\left[\frac{\ln{\left(1+t\right)}}{t}-\frac{\ln{\left(1-2t\cos{\left(\pi-\omega\right)}+t^{2}\right)}}{2t}\right].\\ \end{align}$$
Set $r:=\frac{1-c}{1+c}\in(-1,1)\land\theta:=\pi-\omega\in(0,\pi)$. Then,
$$\begin{align} \mathcal{J}{\left(c,z\right)} &=\int_{\frac{1-c}{1+c}}^{1}\mathrm{d}t\,\left[\frac{\ln{\left(1+t\right)}}{t}-\frac{\ln{\left(1-2t\cos{\left(\pi-\omega\right)}+t^{2}\right)}}{2t}\right]\\ &=\int_{r}^{1}\mathrm{d}t\,\left[\frac{\ln{\left(1+t\right)}}{t}-\frac{\ln{\left(1-2t\cos{\left(\theta\right)}+t^{2}\right)}}{2t}\right]\\ &=\int_{r}^{1}\mathrm{d}t\,\frac{\ln{\left(1+t\right)}}{t}-\int_{r}^{1}\mathrm{d}t\,\frac{\ln{\left(1-2t\cos{\left(\theta\right)}+t^{2}\right)}}{2t}\\ &=\int_{-1}^{-r}\mathrm{d}u\,\frac{(-1)\ln{\left(1-u\right)}}{u};~~~\small{\left[t=-u\right]}\\ &~~~~~-\frac12\int_{0}^{1}\mathrm{d}t\,\frac{\ln{\left(1-2t\cos{\left(\theta\right)}+t^{2}\right)}}{t}+\frac12\int_{0}^{r}\mathrm{d}t\,\frac{\ln{\left(1-2t\cos{\left(\theta\right)}+t^{2}\right)}}{t}\\ &=-\operatorname{Li}_{2}{\left(-1\right)}+\operatorname{Li}_{2}{\left(-r\right)}+\operatorname{Li}_{2}{\left(1,\theta\right)}-\operatorname{Li}_{2}{\left(r,\theta\right)}\\ &=\left(\frac{\pi}{2}-\frac{\theta}{2}\right)^{2}+\operatorname{Li}_{2}{\left(-r\right)}-\operatorname{Li}_{2}{\left(r,\theta\right)}\\ &=\frac14\omega^{2}+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c},\pi-\omega\right)}\\ &=\left[\arctan{\left(z\right)}\right]^{2}+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c},\pi-2\arctan{\left(z\right)}\right)},\\ \end{align}$$
where the dilogarithm and its two-variable variant are given by the integrals
$$\operatorname{Li}_{2}{\left(z\right)}:=-\int_{0}^{z}\mathrm{d}t\,\frac{\ln{\left(1-t\right)}}{t};~~~\small{z\in(-\infty,1]},$$
$$\operatorname{Li}_{2}{\left(r,\theta\right)}:=-\frac12\int_{0}^{r}\mathrm{d}t\,\frac{\ln{\left(1-2t\cos{\left(\theta\right)}+t^{2}\right)}}{t};~~~\small{(r,\theta)\in\mathbb{R}^{2}}.$$
Note: It can be shown that
$$\operatorname{Li}_{2}{\left(1,\theta\right)}=\operatorname{Li}_{2}{\left(-1\right)}+\left(\frac{\pi}{2}-\frac{\theta}{2}\right)^{2};~~~\small{0<\theta<\pi}.$$
Putting everything together, we obtain the following closed form expression for $\mathcal{I}$:
$$\begin{align} \mathcal{I}{\left(a,b\right)} &=\mathcal{J}{\left(c,z\right)}-\mathcal{J}{\left(c^{-1},z\right)}\\ &=\left[\arctan{\left(z\right)}\right]^{2}+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c},\pi-2\arctan{\left(z\right)}\right)}\\ &~~~~~-\left[\arctan{\left(z\right)}\right]^{2}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c}\right)}+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c},\pi-2\arctan{\left(z\right)}\right)}\\ &=\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c}\right)}\\ &~~~~~+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c},\pi-2\arctan{\left(z\right)}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c},\pi-2\arctan{\left(z\right)}\right)}\\ &=\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c}\right)}+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c},\pi-\beta\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c},\pi-\beta\right)}\\ &=\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c}\right)}-\operatorname{Li}_{2}{\left(\frac{1-c}{1+c}\right)}+\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c},\pi-\beta\right)}-\operatorname{Li}_{2}{\left(-\frac{1-c}{1+c},\beta\right)}\\ &=\operatorname{Li}_{2}{\left(\frac{p}{1+\sqrt{1+p^{2}}}\right)}-\operatorname{Li}_{2}{\left(-\frac{p}{1+\sqrt{1+p^{2}}}\right)}\\ &~~~~~+\operatorname{Li}_{2}{\left(\frac{p}{1+\sqrt{1+p^{2}}},\pi-\beta\right)}-\operatorname{Li}_{2}{\left(\frac{p}{1+\sqrt{1+p^{2}}},\beta\right)}\\ &=\operatorname{Li}_{2}{\left(\frac{a}{b+\sqrt{a^{2}+b^{2}}}\right)}-\operatorname{Li}_{2}{\left(-\frac{a}{b+\sqrt{a^{2}+b^{2}}}\right)}\\ &~~~~~+\operatorname{Li}_{2}{\left(\frac{a}{b+\sqrt{a^{2}+b^{2}}},\pi-\arcsin{\left(b\right)}\right)}-\operatorname{Li}_{2}{\left(\frac{a}{b+\sqrt{a^{2}+b^{2}}},\arcsin{\left(b\right)}\right)}.\blacksquare\\ \end{align}$$
