I came across this question in an exercise related to Jensen's Inequality, but I know that the function $ f(x) = \cos x$ changes its concavity after $\frac{\pi}{2}$. So I was not able to prove this result for an obtuse-angled triangle. So I want to know how this question can be solved using Jensen's Inequality. I was able to solve this question using a different approach here:-
Let, $\cos(A) + \cos(B) + \cos(C) = k.$
On factorizing $ \cos A + \cos B $ and applying 2 times angle formula for $\cos C$, we get
$$ 2\cos \left(\frac{A+B}{2}\right)\cos \left(\frac{A-B}{2}\right) + 2\cos^2 \left(\frac{C}{2}\right)-1 = k.$$
As $A+B = \pi - C$, we get:-
$$2\sin \left(\frac{C}{2}\right)\cos \left(\frac{A-B}{2}\right) + 2\cos^2 \left(\frac{C}{2}\right)-1.$$
On changing $2\cos^2 \left(\frac{C}{2}\right)$ into $\sin$ and simplifying, we get a quadratic in $\sin \left(\frac{C}{2}\right)$, Letting $\sin \left(\frac{C}{2}\right)$ as $t$, we get:-
$$t^2 - \left(2\cos \left(\frac{A-B}{2}\right)\right)t + (k-1) = 0.$$
For this to have real roots, its discriminant must be greater than zero, so:
$$ 2k - 2 \le \cos^2 \left(\frac{A-B}{2}\right).$$
Since maximum value of $\cos^2 \left(\frac{A-B}{2}\right)$ is $1$, we get a maximum value of $k = \frac{3}{2}$.