2

Let $n > k$. Let $\textbf{Q} = \textbf{I}_n - \textbf{X}(\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T$ where $\textbf{I}_n$ is an $n \times n$ identity matrix, $\textbf{X}$ is a $n \times (k+1)$ matrix with rank $k+1$ and the first column of $\textbf{X}$ contains only of $1$'s. I would like to show that $\textbf{Q}$ has rank $n-k-1$.

First note that $\textbf{Q}\textbf{X} = \boldsymbol{0}$ and $\textbf{Q}$ is an $n \times n$ matrix. Because $\textbf{Q}\textbf{X} = \boldsymbol{0}$, the rows of $\textbf{Q}$ satisfy the following $k + 1$ constraints: \begin{align*} \sum_{j=1}^nq_{ji} &= 0, \ j = 1, \dotsc, n;\\ \sum_{j=1}^nq_{ji}x_{i1} &= 0, \ j = 1, \dotsc, n;\\ &\vdots\\ \sum_{j=1}^nq_{ji}x_{ik} &= 0, \ j = 1, \dotsc, n. \end{align*}

I feel like I am almost there, but I can't seem to explain why the rank of $\textbf{Q}$ is $n-k-1$ using these constrains.

Zhanxiong
  • 15,126
Vicky
  • 971
  • 7
  • 11
  • 3
    $Q$ is idempotent since the hat matrix $X(X^TX)^{-1}X^T$ is idempotent. And rank of idempotent matrix equals its trace. See https://math.stackexchange.com/q/1582567/321264 – StubbornAtom Jul 27 '22 at 10:18
  • But I dont know the elements of $Q$ on the diagonal, right? – Vicky Jul 29 '22 at 16:38

2 Answers2

2

The matrix $\ X\ $ has $\ k+1\ $ linearly independent columns. Let $\ S\ $ be the set of row vectors orthogonal to the column space of $\ X\ $. Then $\ S\ $ has dimension $\ n-k-1\ $, and for any $\ v\in S\ $, \begin{align} vQ&=v-vX\big(X^TX\big)^{-1}X^T\\ &=v\ , \end{align} so $\ v\ $ is in the row space of $\ Q_\ $. Therefore, the dimension of the row space, and hence the rank of $\ Q\ $ is at least $\ n-k-1\ $.

But $\ QX=0\ $, so the kernel of $\ Q\ $ contains the column space of $\ X\ $ and therefore has dimension at least $\ k+1\ $. By the rank-nullity theorem, therefore, $$ k+1\le\dim\ker Q=n-\text{rank}\,Q\ , $$ giving $\ \text{rank}\,Q\le n-k-1\ $, and hence $\ \text{rank}\,Q= n-k-1\ $.

lonza leggiera
  • 31,984
  • Why would $v$ be in the row space of $Q$? Shouldn't $v$ be transposed? – Vicky Jul 29 '22 at 16:35
  • 1
    $\ v $ is in $\ S\ $, the orthogonal complement of the column space of $\ X\ $. I'm assuming that vectors in the orthogonal complement of a space of column vectors to be already row vectors without transposition. I've now reworded the answer to clarify this. – lonza leggiera Jul 29 '22 at 23:20
  • Which answer are you referring to? The one from River Li? There, I don't understand the inequality $r(Q) \geq n - (k+1)$. Is $r(Q + Y) = n$? Also, why $vX(X^TX)^{-1}X^T = 0$? Is it simply because $v$ is orthogonal to the columns of $X$? – Vicky Jul 30 '22 at 10:34
  • 1
    I'm simply answering your question about $\ v\ $ (which doesn't appear in River Li's answer). In River Li's answer $\ r(Q+Y)=n\ $ because $\ Q+Y=I_n\ $. And yes, if $\ v\ $ is orthogonal to allthe columns of $\ X\ $, then $\ vX=0\ $. – lonza leggiera Jul 30 '22 at 14:24
1

Denote the rank of $A$ by $r(A)$.

Denote $Y = X(X^\mathsf{T} X)^{-1}X^{\mathsf{T}}$. We have $r(Y) = k + 1$.

Using $r(A + B) \le r(A) + r(B)$, we have $$r(Q + Y) \le r(Q) + r(Y)$$ which results in $$r(Q) \ge n - (k + 1).$$

Using $r(AB) \ge r(A) + r(B) - n$, we have $$r(QX) \ge r(Q) + r(X) - n$$ which results in (using $QX = 0$) $$r(Q) \le n - (k + 1).$$

Thus, $r(Q) = n - (k + 1)$.

River Li
  • 49,125