2

I was trying to prove the following:

Let $U\Sigma V$ be the $SVD$ decomposition of $A\in\mathbb R^{m\times n}$, where $\textrm{rank}(A)=k$. Show that the pseudoinverse of $A$ is given by, $$ \displaystyle A^\dagger=\sum_{i=1}^k\sigma_i^{-1}v_iu_i^T. $$

${\bf Proof:}$ Let us show that $AA^\dagger A=A$, $A^\dagger AA^\dagger=A^\dagger$, $(A^\dagger A)^T=A^\dagger A$ and $(AA^\dagger)^T=AA^\dagger$. Particionate $A$ as follows, \begin{align*} \displaystyle A=\left[\begin{array}{c|c} U_1&U_2 \end{array}\right]\left[\begin{array}{c|c} \tilde{\Sigma}&0\\ \hline 0&0 \end{array}\right]\left[\begin{array}{c} V_1^T\\ V_2^T \end{array}\right], \end{align*} where $\tilde{\Sigma}=\textrm{diag}(\sigma_1, \ldots, \sigma_k)$. Then, $A=U_1\tilde{\Sigma}V_1^T$ whereas $A^\dagger=V_1\tilde{\Sigma}^{-1}U_1^T$. Since $V^TV=U^TU=I$ we see $U_1^TU_1=V_1^TV_1=I$. Then, \begin{align*} \displaystyle (A^\dagger A)^T=(V_1\tilde{\Sigma}U_1^T)(U_1\tilde{\Sigma}^{-1}V_1^T)=I=A^\dagger A. \end{align*} Analogously, we see $(AA^\dagger)=I=AA^\dagger$. Using that $A^\dagger A=AA^\dagger=I$ (how can that be? $A$ is not squared) we have, \begin{align*} \displaystyle (A^\dagger AA^\dagger)=(A^\dagger A)A^\dagger=IA^\dagger\ \textrm{and}\ (AA^\dagger A)=A(A^\dagger A)=AI=A. \end{align*}

${\bf Problems:}$ The main problem is the part $(A^\dagger A)=I$ for $A$ is not a squared matrix. Furthermore if that identity held the problem would be almost trivial.. What is the problem with my proof?

Rodrigo de Azevedo
  • 23,223
PtF
  • 9,895
  • A derivation is provided here http://math.stackexchange.com/questions/772039/how-does-the-svd-solve-the-least-squares-problem/2173715#2173715 – dantopa Mar 29 '17 at 16:03

2 Answers2

2

First I guess your SVD decomposition is $A=U\Sigma V^T$ and not $A=U\Sigma V$.

Be carefull of the format of your matrices!

It is true that $V^TV=I_n$ and $U^TU=I_m$ but (unless $n=m$) $I_n\neq I_m$.

Still it is enough to prove that $U_1^T U_1=I_k=V_1^T V_1$.

But then $$\begin{align} A^\dagger\,A & =V_1\tilde \Sigma^{-1} U_1^T U_1\tilde \Sigma V_1^T\\ &=V_1\tilde \Sigma^{-1} I_k\tilde \Sigma V_1^T\\ &=V_1\tilde \Sigma^{-1} \tilde \Sigma V_1^T\\ &=V_1 I_k V_1^T\\ &=V_1 V_1^T\\ &=\left[\begin{array}{c|c} I_k&0\\ \hline 0&0 \end{array}\right] \end{align}\,.$$

And so on.

Sebastien B
  • 954
  • Yes it is $A=U\Sigma V^T$.. But then the proof is not fully wrong? If I fix the order problem it gets right? – PtF Feb 13 '13 at 12:50
  • Well the computation I gave shows that $A^\dagger A\neq I_n$ but $A^\dagger A=\left[\begin{array}{c|c} I_k&0\ \hline 0&0 \end{array}\right]$ (a $n\times n$ matrix). So you have to modify your proof. But the main idea is good to show that it is a pseudo inverse. – Sebastien B Feb 13 '13 at 13:44
  • Ok, I will modify.. Thanks for the help.. – PtF Feb 13 '13 at 13:58
1

You shouldn't assume that $A^\dagger A$ is equal to $I$: \begin{align*} A &= U_1 \widetilde{\Sigma} V_1^T,\\ A^\dagger &= V_1 \widetilde{\Sigma}^{-1} U_1^T,\\ \Rightarrow A^\dagger A &= V_1V_1^T \ \text{ is symmetric}. \end{align*}

user1551
  • 149,263
  • But if $A^\dagger A$ is symmetric then $(A^\dagger A)^T=I$ implies $A^\dagger A=I$ but that is not true..I guess the problem was that I was saying $A^\dagger A$ is the "whole" identity $I_{n\times n}$ but its an identity of smaller order as @Sebastien B shows below.. – PtF Feb 13 '13 at 12:48
  • 1
    @Dion I don't understand why you insist that $A^\dagger A=I$. Consider, for instance, $V_1=\begin{pmatrix}1\0\end{pmatrix}$. Then $V_1V_1^T=\begin{pmatrix}1&0\0&0\end{pmatrix}$ is symmetric, but it is not equal to $I$. In fact, unless $k=n$, $V_1V_1^T$ is not invertible and hence cannot be equal to $I$. – user1551 Feb 13 '13 at 13:04
  • I understand now.. Thanks – PtF Feb 13 '13 at 13:58