34

We know that a square matrix is a covariance matrix of some random vector if and only if it is symmetric and positive semi-definite (see Covariance matrix). We also know that every symmetric positive definite matrix is invertible (see Positive definite). It seems that the inverse of a covariance matrix sometimes does not exist.

Does the inverse of a covariance matrix exist if and only if the covariance matrix is positive definite? How can I intuitively understand the situation when the inverse of a covariance matrix does not exist (does it mean that some of the random variables of the random vector are equal to a constant almost surely)?

Any help will be much appreciated!

Cm7F7Bb
  • 17,879
  • 5
  • 43
  • 69
  • 4
    I suspect the inverse does not exist if and only if $P(X\in H)=1$ for some $H\subset\mathbb R^n$ with - let's say - having a dimension less than $n$. Here $H$ is not requested to be a hyperplane. E.g. also a sphere will do. Pure intuition, though. I could be wrong in this. – drhab Oct 14 '15 at 09:05

4 Answers4

30

If the covariance matrix is not positive definite, we have some $a \in \mathbf R^n \setminus \{0\}$ with $\def\C{\mathop{\rm Cov}}\C(X)a = 0$. Hence \begin{align*} 0 &= a^t \C(X)a\\ &= \sum_{ij} a_j \C(X_i, X_j) a_i\\ &= \mathop{\rm Var}\left(\sum_i a_i X_i\right) \end{align*} So there is some linear combination of the $X_i$ which has zero variance and hence is constant, say equal to $\alpha$, almost surely. Letting $H := \{x \in \mathbf{R}^n: \sum_{i} a_i x_i = \alpha\}$, this means, as @drhab wrote $\mathbf P(X \in H) = 1$ for the hyperplane $H$.

martini
  • 86,011
  • Didn't see your answer when I was posting mine -- hence the apparent duplicacy. – uniquesolution Oct 14 '15 at 09:39
  • No problem ... ${}$ – martini Oct 14 '15 at 09:39
  • Too late: I meant: "I have grown wiser." Now also when it concerns English language. – drhab Oct 14 '15 at 09:57
  • @martini Nice answer (+1)! I'd like to clarify a few details. (a) If $\operatorname{Cov}X$ is not invertible, then there exists $a$ such that $\operatorname{Var}(a^TX)=0$. But this $a$ might not be unique, right? So can we conclude that $\Pr{X\in H}=1$ if $a$ is not unique? (b) Is it also true that $\operatorname{Cov}X$ is not invertible if and only if there exists $a$ such that $\operatorname{Var}(a^TX)=0$? – Cm7F7Bb Oct 15 '15 at 07:17
  • Right, it may not unique, but we can conclude that. In the case where $a$ is not unique (even not up to a scalar factor), we even have that $X$ is almost surely contained in an $<(n-1)$-dimensional subspace (the one orthogonal to all $a$ having $\operatorname{Var}(a^t X) = 0$). Yes that's true. As $\operatorname{Var}(a^t X) = a^t \operatorname{Cov}(X)a$. – martini Oct 15 '15 at 07:20
11

As is nicely explained here What's the best way to think about the covariance matrix?

if $A$ is the covariance matrix of some random vector $X\in\mathbb{R}^n$, then for every fixed $\beta\in\mathbb{R}^n$, the variance of the inner product $\langle\beta,X\rangle$ is given by $\langle A\beta,\beta\rangle$. Now, if $A$ is not invertible, there exists a non-zero vector $\beta\neq 0$ such that $A\beta=0$, and so $\langle A\beta,\beta\rangle = 0$, which means that the variance of $\langle X,\beta\rangle$ is zero.

Proposition 1. If the covariance matrix of a random vector $X$ is not invertible then there exists a non-trivial linear combination of the components of $X$ whose variance is zero.

This is closely related to what drhab mentioned in a comment above - for if the variance of $\langle X,\beta\rangle$ is zero, then $X-a\beta$ is almost surely orthogonal to $\beta$, for some constant $a$.In fact an alternative but equivalent formulation to the proposition above is:

Proposition 2. If the covariance matrix of a random vector $X$ is not invertible then there exists $\beta\neq 0$ such that a translate of $X$ is orthogonal to $\beta$ with probability one.

Community
  • 1
uniquesolution
  • 18,982
  • Too late: I meant: "I have grown wiser." Now also when it concerns English language. – drhab Oct 14 '15 at 09:57
  • Nice answer (+1)! I just want to clarify the details. If $\operatorname{Var}\langle X,\beta\rangle=0$ with $\beta\ne0$, then we have that $\langle A\beta,\beta\rangle=0$. How does it follow that $A\beta=0$, which means that $A$ is not invertible? Also, if $\operatorname{Var}\langle X,\beta\rangle=0$, then it means that $\langle X,\beta\rangle=c$ almost surely with $c\in\mathbb R$ and the vector that is almost surely orthogonal to $X$ is $c\beta$, right? – Cm7F7Bb Oct 14 '15 at 13:22
  • @V.C You are right -- the original formulation was not correct. I edited it. – uniquesolution Oct 14 '15 at 13:57
  • @uniquesolution I'm not saying that something is not correct, I'm just trying to understand the details. Your previous Proposition 1 was so much nicer... Is it possible to show that the covariance matrix is not invertible if there exists such linear combination? For any $X$ and $Y$ in any Hilbert space, there always exists a constant $a$ such that $\langle X-aY,Y\rangle=0$. Take $a=\langle X,Y\rangle/\langle Y,Y\rangle$. Is that right? – Cm7F7Bb Oct 14 '15 at 15:49
  • Yes, the previous answer was much nicer, but the proof I presented was not good enough. Perhaps it is true. I will think about it some more. And as for $a$, you got it right. – uniquesolution Oct 14 '15 at 18:00
  • @uniquesolution What I mean is that you don't need the variance of $\langle X,\beta\rangle$ to be zero to have $\langle X-a\beta,\beta\rangle=0$ almost surely for some constant $a$. – Cm7F7Bb Oct 14 '15 at 19:46
2

The question actually has little to do with probability theory, the observation holds for any square matrix regardless of it's origin.

It's easy to prove by considering the eigenvalues of the matrix. If and only if all of them are non-zero is the matrix invertible. It follows from the characteristic equation $\det (A-\lambda I)=0$, if $\lambda = 0$ is a solution then and only then $\det(A-0I) = \det(A) = 0$.

Positive definite means that all eigenvalues are positive, but positive semi-definite means only that they are non-negative.

skyking
  • 17,080
1

This are some thoughts. Let $x$ be a random vector whose entries are i.i.d. Let $A$ be any square matrix which is not full rank. Then the covariance matrix of the random vector $y=Ax$ is not invertible. To see this, note that $E[Axx^TA]=AE[xx^T]A^T$. Thus, regardless of the rank of $E[xx^T]$, covariance matrix of $y$ will not be invertible.

dineshdileep
  • 9,113