We have to evaluate the following integral:
$$\int \frac{\sqrt{\sin x}}{\sqrt{\sin x} + \sqrt{\cos x}} dx$$
I tried this:
I multiplied both the numerator and denominator by $\sec x$
And substituted $\tan x = t$.
But after that I got stuck.
The book where this is taken from gives the following as the answer: $$\ln(1+t)-\frac14\ln(1+t^4)+\frac1{2\sqrt2}\ln\frac{t^2-\sqrt2t+1}{t^2+\sqrt2t+1}-\frac12\tan^{-1}t^2+c$$ where $t=\sqrt{\cot x}$
$\displaystyle \mathcal{I} = \int\frac{\sqrt{\sin x}}{\sqrt{\sin x}+\sqrt{\cos x}}dx = \int \frac{\sqrt{\tan x}}{1+\sqrt{\tan x}}dx$
substitute $\tan x= t^2$ and $\displaystyle dx = \frac{1}{1+t^4}dt$
$\displaystyle \mathcal{I}= \int\frac{t}{(1+t)(1+t^4)}dt = \frac{1}{2}\int\frac{\bigg((1+t^4)+(1-t^4)\bigg)t}{(1+t)(1+t^4)}dt$
$\displaystyle = \frac{1}{2}\int\frac{t}{1+t}dt+\frac{1}{2}\int\frac{(t-t^2)(1+t^2)}{1+t^4}dt$
$\displaystyle = \frac{1}{2}\int \frac{(1+t)-1}{1+t}dt+\frac{1}{2}\int \frac{t+t^3-(t^2-1)-t^4-1}{1+t^4}dt$
$\displaystyle =-\frac{t}{2}+\frac{1}{2}\ln|t+1|+\frac{1}{4}\int\frac{2t}{1+t^4}+\frac{1}{2}\int\frac{t^3}{1+t^4}dt-\frac{1}{2}\int \frac{t^2-1}{1+t^4}dt-\frac{1}{2}t+\mathcal{C}$
all integrals are easy except $\displaystyle \mathcal{J} = \int\frac{t^2-1}{1+t^4}dt = \int\frac{1-t^{-2}}{\left(t+t^{-1}\right)^2-2}dt = \int\frac{(t-t^{-1})'}{(t-t^{-1})^2-2}dt$
We rationalise the denominator to get $$I =\int \frac {\sin x-\sqrt {\cos x\sin x}}{\sin x-\cos x} dx $$ Writing everything in terms of $\cot x $, we get $$I =\int \mathrm{csc}^2 x\left(\frac {\sqrt {\cot x}-1}{\cot^3 x-\cot^2 x+\cot x-1}\right) dx $$ Now substituting $u=\cot x $ and further $v=\sqrt {u} $ gives us $$I = -\int\frac {2v }{v^5+ v^4 + v+1} dv $$ Performing a partial fraction decomposition we have $$I = \frac {2}{4-4\sqrt {2}}\int \frac {v +\sqrt {2}-1}{v^2 +\sqrt {2}v +1} dv +\frac {2}{4+4\sqrt {2}}\int \frac {v-\sqrt {2}-1}{v^2-\sqrt {2 }v+1} dv+\int \frac {1}{1+v} dv =I_1 +I_2 +I_3$$ Hope you can take it from here.
HINT: Multiply the numerator and the denominator by $\frac{1}{\sqrt{\sin(x)}}$, then $t = \sqrt{\cot(x)} $ after all you'll have $\displaystyle\frac{2t}{(t^4+1)(t+1)}$.
For the sake of exploring other options, consider the substitution
$$y = \frac{\sqrt{\sin x}}{\sqrt{\sin x}+\sqrt{\cos x}} = \frac{\sqrt{\tan x}}{\sqrt{\tan x}+1} \implies x = \arctan \frac{y^2}{(1-y)^2}$$
so $0<y<1$; then
$$\int \frac{\sqrt{\sin x}}{\sqrt{\sin x}+\sqrt{\cos x}} \, dx = \int \frac{2(1-y)y^2}{1-4y+6y^2-4y^3+2y^4} \, dy$$
The denominator can be expressed as a difference of squares and subsequently as a product of quadratic polynomials with real coefficients,
$$\begin{align*} & 1-4y+6y^2-4y^3+2y^4 \\ &= (1-y)^4 + y^4 \\ &= (1-y)^4 \color{red}{+ 2(1-y)^2y^2} + y^4 \color{red}{- 2(1-y)^2y^2} \\ &= \left[(1-y)^2 + y^2\right]^2 - 2(1-y)^2y^2 \\ &= \left((1-y)^2 + y^2 \color{red}+ \sqrt2\,(1-y)y\right) \left((1-y)^2 + y^2 \color{red}- \sqrt2\,(1-y)y\right) \\ &= \left(1 - (2-\sqrt2)y + (2-\sqrt2)y^2\right) \left(1 - (2+\sqrt2)y + (2+\sqrt2)y^2\right) \end{align*}$$
and hence the integrand can be decomposed into
$$\frac1{\sqrt2} \left[\frac y{1 - (2+\sqrt2)y + (2+\sqrt2)y^2} - \frac y{1 - (2-\sqrt2)y + (2-\sqrt2)y^2} \right]$$
For $0<a<4$ (which contains $2\pm\sqrt2$), we may use the elementary result
$$\begin{align*} & \int \frac{y}{1-ay+ay^2} \, dy \\ &= \frac1{\sqrt{4a-a^2}} \arctan \left(\sqrt{\frac a{4-a}} (2y-1)\right) + \frac1{2a} \log\left\lvert1-ay+ay^2\right\rvert + C \end{align*}$$
$$\displaystyle\int\frac{\sqrt{\sin x}dx}{\sqrt{\sin x}+\sqrt{\cos x}}\;=\;\int\frac{dx}{1+\sqrt{\cot x}},\\ \displaystyle\cot x=z^{2} \;\;\Rightarrow\;\; -(1+z^{2})dx=2zdz,\\ \displaystyle\int\frac{dx}{1+\sqrt{\cot x}} \;=\; -\int\frac{2zdz}{(1+z^{2})(1+z)} \;=\; -2\int\frac{z(z-1)dz}{z^{4}-1}.\\ \displaystyle\frac{z}{z^{4}-1}\;=\;\frac{1}{2}\left( \frac{z}{z^{2}-1}-\frac{z}{z^{2}+1} \right),\\ \displaystyle\frac{z^{2}}{z^{4}-1}\;=\;\frac{1}{2}(\frac{1}{z^{2}-1}+\frac{1}{z^{2}+1}),\\ \displaystyle\frac{z^{2}-z}{z^{4}-1}=\frac{1}{2}(\frac{1-z}{z^{2}-1})+\frac{1}{2}(\frac{1}{z^{2}+1}+\frac{z}{z^{2}+1})=-\frac{1}{2(z+1)}+\frac{1}{2}(\frac{1}{z^{2}+1}+\frac{z}{z^{2}+1}).\\ \displaystyle-2\int\frac{z(z-1)dz}{z^{4}-1}\;=\;\ln\left| z+1 \right|-\arctan{z}-\ln\sqrt{z^{2}+1}+c,\\ \displaystyle\int\frac{\sqrt{\sin x}dx}{\sqrt{\sin x}+\sqrt{\cos x}}\;=\;\ln\left(\frac{\sqrt{\cot{x}}+1}{\sqrt{\cot{x}+1}} \right)-\arctan{\sqrt{\cot{x}}}+c.$$
