14

Let $X_1, X_2$ be jointly normal $N(\mu, \Sigma)$.

I know that in general, $\mathbb{E}[X_2|X_1]$ can be computed by integrating the conditional density, but in the case of jointly normal variables, it suffices to do a linear projection:

$\mathbb{E}[X_2 | \sigma(X_1)] = \mathbb{E}[X_2|\mathrm{span}(\mathbf{1}, X_1)] = \mu_2 + \frac{\mathrm{cov}(X_2, X_1)}{\mathrm{var}(X_1)} (X_1 - \mu_1) $

Is there a neat proof of this fact (one doesn't require doing any integrals)? Looking for references too.

V. Vancak
  • 16,927
user357269
  • 765

3 Answers3

20

I've found an answer that I'm happy with:

$$ Y - \frac{\mathrm{cov}(X, Y)}{\mathrm{var}(X)} X $$ is jointly normal with $X$ and uncorrelated, hence independent.

Therefore

$$\mathbb{E}[Y|X] = \mathbb{E}[Y - \frac{\mathrm{cov}(X, Y)}{\mathrm{var}(X)} X + \frac{\mathrm{cov}(X, Y)}{\mathrm{var}(X)} X| X] \\ = \mathbb{E}[Y - \frac{\mathrm{cov}(X, Y)}{\mathrm{var}(X)} X] +\frac{\mathrm{cov}(X, Y)}{\mathrm{var}(X)}X \\ = \mathbb{E} [Y] + \frac{\mathrm{cov}(X, Y)}{\mathrm{var}(X)} (X - \mathbb{E}[X])$$

user357269
  • 765
  • But why $Y-Xcov(X,Y)/var(X)$ is uncorrelated with X? – clement Apr 09 '19 at 00:23
  • 1
    @YibeiHe: you can use bilinearity of $\mathrm{cov}$ to show their covariance comes out to $0$ – user357269 Apr 10 '19 at 13:11
  • 2
    This is the solution when $(X,Y)$ have bivariate normal distribution. For the general multivariate normal case, where $X$ and $Y$ are jointly normal random vectors, the formula is analogous, with $\operatorname{Cov}(X,Y)/\operatorname{Var}(X)$ replaced by the product of the cross-covariance matrix of $X, Y$ and the inverse of the covariance matrix of $X$. For a derivation see https://math.stackexchange.com/a/2209484/215011 – grand_chat Mar 10 '21 at 22:51
7

Let $(X_1, X_2) \sim MVN (\mu, \Sigma)$, then recall that \begin{align} f_{X_2|X_1}(x_2) = \frac{f_{X_1, X_2}(x_1, x_2)}{f_{X_1}(x_1)} \, , \end{align} where $$ f_{X_1,X_2}(x_1,x_2)=\\ \frac{1}{2\pi \sigma_1\sigma_2 \sqrt{1-\rho}}\exp\left( - \frac{1}{2(1-\rho^2)}\left(\frac{(x_1 - \mu_1)^2}{\sigma^2} + \frac{(x_2 - \mu_2)^2}{\sigma^2} - \frac{2\rho(x_1-\mu_1)(x_2 - \mu_2)}{\sigma_1\sigma_2} \right) \right), $$ and $$ f_{X_1}(x) = \frac{1}{\sqrt{2\pi\sigma_1^2}}\exp\left( - \frac{(x_1-\mu_1)^2}{2\sigma_1^2} \right). $$

After some simple algebra and rearrengments you'll find that $$ X_2|X_1 = x_1 \sim N\left( \mu_2 + \rho\frac{\sigma_2}{\sigma_1}(x_1 - \mu_1), (1- \rho^2)\sigma_2^2 \right). $$

V. Vancak
  • 16,927
0

Just as a complement to @user357269's answer, I will mainly focus on explaining why we have the coefficient $\frac{cov(X_1,X_2)}{Var(X_1)}$.

It starts by acknowledging the following result about Bivariate normal: $(X_1,X_2)^T \sim N(\mu,\Sigma) \Rightarrow \exists$ a lower triangular matrix $A \in \mathbb{R}^{2 \times 2}$, and $Z_1, Z_2 \sim N(0,1) \; iid$ such that $(X_1,X_2)^T = A (Z_1,Z_2)^T + \mu$.

Furthermore, we have $AA^T = \Sigma =\pmatrix{Var(X_1) & cov(X_1,X_2)\\cov(X_1,X_2) & Var(X_2)}$.

Using Cholesky decomposition to solve A, we get $A=\pmatrix{\sqrt{Var(X_1)} & 0\\ \frac{cov(X_1,X_2)}{\sqrt{Var(X_1)}} & \sqrt{Var(X_2)-\frac{cov(X_1,X_2)^2}{Var(X_1)}}}$.

So, we can express $X_1 = \sqrt{Var(X_1)}Z_1 + \mu_1$ and $X_2=\frac{cov(X_1,X_2)}{\sqrt{Var(X_1)}} Z_1 + \sqrt{Var(X_2)-\frac{cov(X_1,X_2)^2}{Var(X_1)}} Z_2 + \mu_2$.

Since, $Z_1 \perp Z_2$, calculating the conditional expectation becomes much easier: \begin{align*} &E[X_2|X_1]\\ &=E[\frac{cov(X_1,X_2)}{\sqrt{Var(X_1)}} Z_1 + \sqrt{Var(X_2)-\frac{cov(X_1,X_2)^2}{Var(X_1)}} Z_2 + \mu_2|\sqrt{Var(X_1)} Z_1+ \mu_1]\\ &=E[\frac{cov(X_1,X_2)}{\sqrt{Var(X_1)}} Z_1 + \sqrt{Var(X_2)-\frac{cov(X_1,X_2)^2}{Var(X_1)}} Z_2 + \mu_2|Z_1]\\ &=\frac{cov(X_1,X_2)}{\sqrt{Var(X_1)}} Z_1 + \mu_2 \\ &= \frac{cov(X_1,X_2)}{Var(X_1)}(X_1 - \mu_1) + \mu_2 \end{align*}

Jake ZHANG Shiyu
  • 409