I need to find a good contour for $\int_0^{2\pi}\frac{\sin^2(x)}{a + b\cos(x)}\ dx$ but I don't know which one to choose. Both a semicircular, and rectangular contour look ugly for this.
I've been looking at a semicircular contour of radius $2\pi$, but then I have the problem that I don't know whether the singularity is inside or outside the closed region.
If it helps, the answer is $\frac{2\pi}{b^2}\left[a - \sqrt{a^2 - b^2}\right]$
Note that the integral diverges for $a\le b$. Therefore, we assume throughout the development that $a>b$.
We can simplify the task by rewriting the integrand as
$$\begin{align} \frac{\sin^2(x)}{a+b\cos(x)}&=\frac{a}{b^2}-\frac{1}{b}\cos(x)-\left(\frac{a^2-b^2}{b^2}\right)\frac{1}{a+b\cos(x)} \end{align}$$
Then, the integral of interest reduces to
$$\int_0^{2\pi}\frac{\sin^2(x)}{a+b\cos(x)}\,dx=\frac{2\pi a}{b^2}-\frac{a^2-b^2}{b^2}\int_0^{2\pi}\frac{1}{a+b\cos(x)}\,dx \tag 1$$
We enforce the substitution $z=e^{i x}$ in the integral on the right-hand side of $(1)$ and obtain
$$\begin{align} \int_0^{2\pi}\frac{1}{a+b\cos(x)}\,dx& =\oint_{|z|=1}\frac{1}{a+b\left(\frac{z+z^{-1}}{2}\right)}\frac{1}{iz}\,dz\\\\ &=\frac2{ib}\oint_{|z|=1}\frac{1}{(z+(a/b)-\sqrt{(a/b)^2-1})(z+(a/b)+\sqrt{(a/b)^2-1})}\,dz\\\\ &=2\pi i \frac2{ib} \frac{1}{2\sqrt{(a/b)^2-1}}\\\\ &=\frac{2\pi}{\sqrt{a^2-b^2}} \end{align}$$
Putting it all together, the integral of interest is
$$\bbox[5px,border:2px solid #C0A000]{\int_0^{2\pi}\frac{\sin^2(x)}{a+b\cos(x)}\,dx=\frac{2\pi}{b^2}\left(a-\sqrt{a^2-b^2}\right)}$$
as was to be shown!
Here is an approach where we do not simplify the integral before switching to complex variables.
Suppose we are looking to evaluate $$\int_0^{2\pi} \frac{\sin^2 x}{a+b\cos x} dx.$$
Put $z=e^{ix}$ so that $dz = i e^{ix} \; dx = iz \; dx$ and use $$\cos x = \frac{e^{ix}+e^{-ix}}{2} \quad\text{and}\quad \sin x = \frac{e^{ix}-e^{-ix}}{2i}$$
to get $$- \int_{|z|=1} \frac{(z-1/z)^2/4}{a+b(z+1/z)/2} \frac{1}{iz} dz = i \int_{|z|=1} \frac{(z-1/z)^2/4}{bz^2/2 + az + b/2} \; dz \\ = i \int_{|z|=1} \frac{1}{4z^2} \frac{z^4-2z^2+1}{bz^2/2 + az + b/2} \; dz.$$
The two poles here are at $$\rho_{1,2} = \frac{-a\pm\sqrt{a^2-b^2}}{b} = \frac{a}{b} \left(-1\pm\sqrt{1-b^2/a^2}\right).$$
There is another pole at $z=0$ with residue $$\frac{1}{4} \left.\frac{4z^3-4z}{bz^2/2 + az + b/2} - \frac{1}{4} \frac{z^4-2z^2+1}{(bz^2/2 + az + b/2)^2} (bz+a)\right|_{z=0} = - \frac{a}{b^2}.$$
Now there are several possibilities here, we discuss two of them.
First scenario. Suppose $a$ and $b$ are real and $a\le b.$ We get
$$|\rho_{1,2}| =\left| \frac{-a\pm i\sqrt{b^2-a^2}}{b}\right| = \sqrt{\frac{a^2}{b^2}+\frac{b^2-a^2}{b^2}} = 1.$$
This means the two poles are located precisely on the circular contour so the best we can hope for is to get the Cauchy Principal Value of the integral. These are simple poles so the contribution is half the residue for both poles. We now compute these residues.
$$\mathrm{Res}_{z=\rho_{1,2}} \frac{1}{4z^2} \frac{z^4-2z^2+1}{bz^2/2 + az + b/2} = \frac{(\rho_{1,2}^2-1)^2}{4\rho_{1,2}^2} \frac{1}{b\rho_{1,2} + a}.$$
Now we have by definition that $$\rho_{1,2}^2 = -2a\rho_{1,2}/b - 1 \quad\text{and hence}\quad \frac{\rho_{1,2}^2-1}{\rho_{1,2}} = -2\frac{a}{b} - \frac{2}{\rho_{1,2}}.$$
Furthermore $$- \frac{1}{\rho_{1,2}} = 2\frac{a}{b} + \rho_{1,2} \quad\text{and therefore}\quad \frac{\rho_{1,2}^2-1}{\rho_{1,2}} = 2\frac{a}{b} + 2\rho_{1,2}.$$
Square this to get
$$4\frac{a^2}{b^2} + 8 \frac{a}{b}\rho_{1,2} + 4\rho_{1,2}^2 \\ = 4\frac{a^2}{b^2} + 8 \frac{a}{b}\rho_{1,2} -8\frac{a}{b}\rho_{1,2} - 4 = 4\frac{a^2}{b^2} - 4.$$
This finally yields for the two residues
$$\left(\frac{a^2}{b^2} - 1\right)\frac{1}{b\rho_{1,2}+a} = \pm \frac{a^2-b^2}{b^2} \frac{1}{\sqrt{a^2-b^2}} = \pm \frac{\sqrt{a^2-b^2}}{b^2}.$$
Recall that we are currently evaluating the case where both poles are on the contour. We thus get for the value
$$i\times 2\pi i \times \left(-\frac{a}{b^2} + \frac{1}{2} \frac{\sqrt{a^2-b^2}}{b^2} - \frac{1}{2} \frac{\sqrt{a^2-b^2}}{b^2}\right) = \frac{2\pi a}{b^2}.$$
Second scenario. On the other hand when $a\gt b$ we put $b/a=q$ with $0\lt q\lt 1$ a positive real number. We get
$$\rho_{1,2} = \frac{1}{q} \left(-1 \pm \sqrt{1-q^2}\right).$$
Note however that with the positive determination of the square root that we have been using we have that $-1-\sqrt{1-q^2} \lt -1$ and $1/q \gt 1$ so their product is a negative real less than $-1.$ Hence $\rho_2$ is not inside the contour. Now
$$\rho_1\rho_2 = \frac{(-a)^2-(a^2-b^2)}{b^2} = 1$$
and hence $\rho_1$ is the inverse of $\rho_2.$ With $|\rho_2|>1$ we get $|\rho_1|<1.$ Therefore the pole at $\rho_1$ is the only one of the two inside the contour and we get
$$i\times 2\pi i \times \left(-\frac{a}{b^2} + \frac{\sqrt{a^2-b^2}}{b^2}\right) = \frac{2\pi}{b^2} \left(a-\sqrt{a^2-b^2}\right).$$
Additional scenarios are left to the reader.
