Completing the square in $N$ dimensions

Question

This is very important for Bayesian methods in statistics, but I haven't been able to find a reference which specifically touches on my situation.

Assume all matrices and vectors below are matrices and vectors with all real entries.

All boldface lowercase letters are column vectors, and boldface uppercase letters are matrices (including the lowercase and uppercase Greek letters). In the below, $\mathbf{y}$, $\boldsymbol\theta$ (with any subscripts), and $\boldsymbol\mu$ are column vectors; $\mathbf{X}$, $\boldsymbol\Sigma$ (with any subscripts) are matrices; and $c$ is a scalar in $\mathbb{R}$.

$\mathbf{X}^{\prime}$ denotes the transpose of a matrix $\mathbf{X}$.

I have a sum of two quadratic forms $$(\mathbf{y}-\mathbf{X}\boldsymbol\theta)^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}(\mathbf{y}-\mathbf{X}\boldsymbol\theta) + (\boldsymbol\theta-\boldsymbol\theta_0)^{\prime}\boldsymbol\Sigma^{-1}_0(\boldsymbol\theta-\boldsymbol\theta_0)\tag{1}$$ which I would like to write in the form $$(\boldsymbol\theta-\boldsymbol\mu)^{\prime}\boldsymbol\Sigma^{-1}(\boldsymbol\theta-\boldsymbol\mu) + c\tag{2}$$ I don't care what $c$ is. What I'm mainly interested in is what $\boldsymbol\mu$ and $\boldsymbol\Sigma$ are.

After a lot of work, I was able to write the parts of $(1)$ which depend on $\boldsymbol\theta$ in the following form (remember, I don't care about $c$): $$\boldsymbol\theta^{\prime}(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma_0^{-1})\boldsymbol\theta-\boldsymbol\theta^{\prime}(\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y}+\boldsymbol\Sigma_0^{-1}\boldsymbol\theta_0)-(\mathbf{y}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{X}+\boldsymbol\theta_0^{\prime}\boldsymbol\Sigma_0^{-1})\boldsymbol\theta\tag{3}$$ You should assume that all $\boldsymbol\Sigma$ matrices, regardless of the subscript, are symmetric and positive definite.

How can I write $(3)$ in the form $(2)$? I would really appreciate an explanation of how the procedure works in general.

Answer 1

The trick is to expand $(2)$, taking advantage of the fact that all the $\boldsymbol{\Sigma}$ matrices are symmetric (so that their inverses will also be symmetric, and exist, as they are positive definite):- $$(\boldsymbol\theta-\boldsymbol\mu)^{\prime}\boldsymbol\Sigma^{-1}(\boldsymbol\theta-\boldsymbol\mu) + c=\boldsymbol{\theta}^{\prime}\color{red}{\boldsymbol\Sigma^{-1}}\boldsymbol\theta-2\boldsymbol\theta^{\prime}\color{blue}{\boldsymbol\Sigma^{-1}\boldsymbol\mu}+\boldsymbol\mu^{\prime}\boldsymbol\Sigma^{-1}\boldsymbol\mu+c $$ Note that term $\boldsymbol\mu^{\prime}\boldsymbol\Sigma^{-1}\boldsymbol\mu$ is a constant term, insofar it has no dependence on $\boldsymbol\theta$ and can be absorbed into $c$.

Re-using the symmetric property of all the $\boldsymbol{\Sigma}$ matrices, we can further simplify equation $(3)$ as follows:- $$\boldsymbol\theta^{\prime}\color{red}{(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma_0^{-1})}\boldsymbol\theta-2\boldsymbol\theta^{\prime}\color{blue}{(\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y}+\boldsymbol\Sigma_0^{-1}\boldsymbol\theta_0)}$$

Comparing both equations, we have $$\begin{align}\boldsymbol\Sigma^{-1}=(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma_0^{-1}) \\ \boldsymbol\Sigma^{-1}\boldsymbol\mu=(\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y}+\boldsymbol\Sigma_0^{-1}\boldsymbol\theta_0)\end{align}$$ leading to $$\begin{align}\boldsymbol\Sigma&=(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma_0^{-1})^{-1} \\ \boldsymbol\mu &=(\mathbf{X}^{\prime}\boldsymbol\Sigma^{-1}_{\boldsymbol\eta}\mathbf{X}+\boldsymbol\Sigma_0^{-1})^{-1}(\mathbf{X}^{\prime}\boldsymbol\Sigma_{\boldsymbol\eta}^{-1}\mathbf{y}+\boldsymbol\Sigma_0^{-1}\boldsymbol\theta_0)\end{align}$$ In this case, the constant term $c$ - which you do not care about - will simply be $-\boldsymbol\mu^{\prime}\boldsymbol\Sigma^{-1}\boldsymbol\mu$.