17

Suppose $A$ is fat(number of columns > number of rows) and full row rank. The projection of $z$ onto $\{x\mid Ax = b\}$ is (affine)

$$P(z) = z - A^T(AA^T)^{-1}(Az-b)$$

How to show this?

Note: $A^T(AA^T)^{-1}$ is the pseudo-inverse of $A$

What I am thinking is from:

  1. Least square problem: $$\text{min $\left\|\: Ax-b \,\right\|_2$}$$
    The solution for this is $\hat{x} = A^T(AA^T)^{-1}b$. It seems $(Az - b )$ above is the role of $b$ here.

  2. Vector projection of $x$ onto $y$: $$p = \frac{x^Ty}{y^Ty}y$$

But I still cannot figure out how to prove the above result.

Royi
  • 10,050
sleeve chen
  • 8,576

1 Answers1

21

The projection of $z$ onto the set $\{x:\ Ax=b\}$ is given by the solution of $$ \min \frac12\|x-z\|^2 \quad \text{ subject to } Ax=b. $$ The KKT system is a necessary (since constraints are linear) and sufficient (since this is a convex problem): $$ Ax=b, \quad x-z +A^T\lambda = 0. $$ Multiply the second equation by $A$, solve for $\lambda$: $\lambda=(AA^T)^{-1}(Az-b)$, plug this again into the second equation to obtain $$ x = z - A^T\lambda =z-A^T (AA^T)^{-1}(Az-b) $$

daw
  • 54,637
  • 2
  • 44
  • 85
  • How to get $x - z + A^T\lambda$? Do you form the Lagrangian and take the gradient of Lagrangian with respect to $x$ and get $x - z + A^T\lambda$? – sleeve chen Jun 10 '15 at 19:47
  • 1
    Exactly, the Lagrangian is $\frac12|x-z|^2 + \lambda^T(Ax-b)$. – daw Jun 10 '15 at 19:48
  • 2
    This also follows from linear algebra, since at the optimum, you have $x-z \bot \ker A$. – copper.hat Jun 10 '15 at 19:57
  • @daw Additional question, you solve $x-z+A^T\lambda=0$ for $\lambda$ instead of $x$ and at last you can find $x$ why? Because of KKT? primal optimal = dual optimal? I am a little bit confused about this. thx! – sleeve chen Jun 10 '15 at 20:14
  • 1
    I use the equation twice: first to get $\lambda$ (while applying $Ax=b$), second to get $x$. – daw Jun 10 '15 at 20:39