I am trying to understand the high-dimensional geometry behind Bayesian estimation.
When you multiply two Normal densities with respective means $\mu_1, \mu_2$ and covariances $\Sigma_1, \Sigma_2$, the renormalized product is again a Normal density $\mathcal N (\mu_3, \Sigma_3)$, as is shown in this post: Product of Two Multivariate Gaussians Distributions, in summary: \begin{align} \Sigma_3 &=(\Sigma_1^{-1}+\Sigma_2^{-1})^{-1}\\ &= \Sigma_1(\Sigma_1 + \Sigma_2)^{-1}\Sigma_2\\ \mu_3 &= \Sigma_2(\Sigma_1 + \Sigma_2)^{-1}\mu_1 + \Sigma_1(\Sigma_1 + \Sigma_2)^{-1}\mu_2\\ &=\Sigma_3(\Sigma_1^{-1}\mu_1 + \Sigma_2^{-1}\mu_2) \end{align}
The formulas are clear, and the intuition of what happens, too:
- The new precision (inverse covariance) is the sum of the old precisions.
- The new mean is a weighted average of the old means.
The thing I cannot grasp is the geometric interpretation of the multivariate weighted average. In 1D, and in general if the eigenvectors of the covariance matrices are collinear, then $\mu_3$ is on the line between $\mu_1$ and $\mu_2$, i.e. $\mu_3 = \alpha\mu_1 + (1-\alpha)\mu_2$, where $\alpha \equiv \alpha(\Sigma_1, \Sigma_2)\in[0,1]$. But what if the eigenvectors are not collinear?
The intuition is that it goes where the contour ellipsoids "overlap". (Sorry, not enough reputation for visualizations, but this blog on Kalman filters does a pretty good job.) But where exactly is that point? Here is what I found:
- It's not intersection of the lines defined by the eigenvectors.
- The second equation for the mean suggests: transfer them into the coordinate system defined by their respective covariance. Add them, then skew the result with the new covariance. I do not really understand what this means geometrically.
- I found that this last point is related to the canonical form of the Normal distribution, in that $\Sigma_i^{-1}\mu_i$ is one of the canonical parameters. But I don't have an interpretation for adding these, let alone multiplying them with the new covariance.
- I could not pin down this concept of "overlapping ellipsoids". Which ellipsoids? Is it an intersection? The mass point of an intersection area? Something completely different?
I feel like I'm missing something obvious. Any help appreciated!
