Consider the state space system given by $$ \dot{x} = Ax(t) + Bu(t)$$ $$ y = Cx(t) + Du(t)$$
The standard LQR cost is given by $$ J = \int \big( x(t)^T Q x(t) + u(t)^T R u(t) \big)dt$$
Instead, consider the cost given by $$ J = \int \big( y(t)^T Q y(t) + u(t)^T R u(t) \big)dt$$
Typically, the states are regulated by designing an observer, but here, the cost is based only on the output not the states. I can't find any references on this scenario, and I am not sure how to solve this problem.
I will base this anwer on the answer by Johan Löfberg to a related question, where I asked how to include costs on the state derivatives $\dot{x}$. The answer is practically the same if you want the output $y$ instead.
You have the the state space equations
$$ \begin{align} \dot{x} &= A x + B u \\ y &= C x + D u \end{align} $$
The general cost function for the linear quadratic regulator (LQR) is
$$ J_1 = \int_0^\infty (x^T Q x + u^T R u + 2 x^T N u) \, dt $$
If you want to put a cost directly on the output $y$ you can do that:
$$ \begin{align} y^T L y &= (C x + D u)^T L (C x + D u) \\ &= x^T (C^T L \,C)x + x^T(C^T L \, D)u + u^T(D^T L \, C)x + u^T(D^T L \, D)u \\ &= x^T (C^T L \,C)x + u^T(D^T L \, D)u + 2 x^T (C^T L \, D) u \end{align} $$
So if you want replace a general cost function which has a cost for $y$ with the standard cost function:
$$ \begin{align} J_2 &= \int_0^\infty (x^T Q x + u^T R u + 2 x^T N u + y^T L y) \, dt \\ &= \int_0^\infty ( x^T \bar{Q} x + u^T \bar{R} u + 2 x^T \bar{N}u ) \, dt \end{align} $$
and the new matrices
$$ \begin{align} \bar{Q} &= Q + C^T L \, C \\ \bar{R} &= R + D^T L \, D \\ \bar{N} &= N + C^T L \, D \end{align} $$
You can then solve the LQR problem with $J_2$ instead of $J_1$ with the standard methods. Of course you must take $Q, R, N, L$ so that $\bar{Q}, \bar{R}, \bar{N}$ satisfy the requirements for LQR.
The nice thing is that if $D = 0$ and you set $N = 0$ that you have the LQR problem without mixed term because then $\bar{N} = 0$. This is good for robustness (see the comments by Kwin van der Veen to this answer by Arastas).
However, you will still need all states for the controller because it is still of the form $u=-K x$.
