5

I have a convex optimization of the form $$ \min_x \frac{1}{2} x^TAx-x^Tb \\ \text{s.t.}\ (I-P)x=0 $$ where $A$ is a $n$ by $n$ positive definite matrix, and $P$ is a $n$ by $n$ projection matrix (it has $p$ eigenvalues equal to zero, and $n-p$ eigenvalues equal to one)

Intuitively, it seems like I can separate the subspaces by rewriting it to: $$ \min_x \frac{1}{2} x^TP^TAPx+\frac{\gamma}{2} x^T(I-P)^T(I-P)x-x^TP^Tb $$ And since $P$ is a projection matrix, it is symmetric and idempotent, it can be simplified to: $$ \min_x \frac{1}{2} x^T(PAP+\gamma(I-P))x-x^TPb $$ This results in an unconstrained optimization, and the result is obtained by solving: $$(PAP+\gamma(I-P))x=Pb$$ This formulation is particularly handy because the matrix $(PAP+\gamma(I-P))$ is still positive definite for any positive $\gamma$ value (which is in fact the eigenvalues of the constraint subspace), so it can be solved numerically using the conjugate gradient method.
In practice, it works very well, but to be honest, I don't know if there is a flaw in my intuition, and I haven't found any resources on the subject.
Is there a known method that I can use to demonstrate my intuition is correct?

Michael M.
  • 159
  • I didn't understand how you obtain $\gamma$ but I think it doesn't matter because $(I-P)x=0$ so that summand becomes $0$. – Lancet S. Jan 11 '23 at 13:26
  • Indeed. This is only added to make the matrix SPD. $\gamma$ is there to scale the eigenvalues for better CG convergence in case a scale factor of 1 increase the condition number of the matrix. – Michael M. Jan 13 '23 at 18:05

1 Answers1

3

First of all,

$$\frac{1}{2}x^{T}Ax-x^{T}b=\frac{1}{2}x^{T}(P+I-P)^{T}A(P+I-P)x-x^{T}(P+I-P)^{T}b. $$

Notice that

$$\frac{1}{2}x^{T}(P+I-P)^{T}A(P+I-P)x=\frac{1}{2}x^{T}P^{T}APx+\frac{1}{2}x^{T}(I-P)^{T}A(I-P)x+\frac{1}{2}x^{T}P^{T}A(I-P)x+\frac{1}{2}x^{T}(I-P)^{T}APx, $$

and

$$x^{T}(P+I-P)^{t}b=x^{t}P^{t}b+x^{t}(I-P)^{t}b. $$

Now with the condition $(I-P)x=0$ we have that,

$$\frac{1}{2}x^{T}Ax-x^{T}b=\frac{1}{2}x^{T}P^{T}APx-x^{T}P^{t}b. $$

The problem is that P is semi-defined positive so $P^{T}AP$ isn't guranteed to be defined positive and thus it isn't guranteed that the minimuns are global and unique.

However, since $f(x)=\frac{1}{2}x^{t}Ax-x^{t}b$ is a convex function, and the restriction set has the form $h(x)=0$ with $h$ affine, the problems belongs to the category of convex optimization problems, that are easily solved by softwares like AMPL.

That problems could be also solved by hand using KKT conditions (becasuse convex optimization problems satisfies the hypothesis of that method).

When the restriction set has complicated functions ($h(x)=cos(x)=0$, for example) a penalization method is usually used. You can search information about exterior penalty function methods or augmented lagrangian method.

Lancet S.
  • 525
  • 1
    Thank you for your answer. As you say, the matrix is not definite positive anymore, so I cannot use a conjugate gradient anymore. However, since $x^T(I-P)x$ is always equal to zero, can I use that term to make the matrix definite positive? – Michael M. Jan 12 '23 at 16:35
  • 1
    I think you could work with the semi-definite matrix (https://scicomp.stackexchange.com/questions/35239/what-happens-when-i-use-a-conjugate-gradient-solver-with-a-symmetric-positive-se). But I think you can solve your problem more easily with KKT conditions. https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions. And a post of mine in which I post a theorem that can be used in your case (https://math.stackexchange.com/questions/3060883/kkt-conditions-equality-constraints/4616405#4616405). I hope it helps! – Lancet S. Jan 12 '23 at 17:14
  • This definitely helps! Thanks a lot, everything makes more sense now. – Michael M. Jan 13 '23 at 07:33