Proof of the independence of the sample mean and sample variance

Question

I've been trying to establish that the sample mean and the sample variance are independent. One motivation is to try and write the sample variance, $S^{2}$ as a function of $\left\{ X_{2}-\bar{X},X_{3}-\bar{X},\cdots,X_{n}-\bar{X}\right\} =A$ only. Then we proceed by showing that $A$ and $\bar{X}$ are independent (which I'm unable to show), which then implies the independence of $S^{2}$ and $\bar{X}.$

I would appreciate it if the good people of M.SE would help guide me in the right direction.
Thanks.

Edit: The random samples $X_1,\cdots,X_n$ are from an $N(\mu, \sigma)$ distribution.

Answer 1

I guess this is probably a little late, but this result is immediate from Basu's Theorem, provided that you are willing to accept that the family of normal distributions with known variance is complete. To apply Basu, fix $\sigma^2$ and consider the family of $N(\mu, \sigma^2)$ for $\mu \in \mathbb R$. Then $\frac{(n - 1)S^2}{\sigma^2} \sim \chi^2_{n - 1}$ so $S^2$ is ancillary, while $\bar X$ is complete sufficient, and hence they are independent for all $\mu$ and our fixed $\sigma^2$. Since $\sigma^2$ was arbitrary, this completes the proof.

This can also be shown directly without too much hassle. One can find the joint pdf of $(A, \bar X)$ directly by making a suitable transformation to the joint pdf of $(X_1,\cdots, X_n)$. The joint pdf of $(A, \bar X)$ factors as required, which gives independence. To see this quickly, without actually doing the transformation, skipping some algebra we may write

$$f(x_1, x_2, ..., x_n) = (2\pi \sigma^2)^{-n/2} \exp\left\{-\frac{\sum(x_i - \bar x)^2}{2\sigma^2}\right\} \exp\left\{-\frac{n(\bar x - \mu)^2}{2\sigma^2}\right\}$$

and we can see that everything except for the last term depends only on $$(x_2 - \bar x, x_3 - \bar x, ..., x_n - \bar x)$$ (note we may retrieve $x_1 - \bar x$ from only the first $n - 1$ deviations) and the last term depends only on $\bar x$. The transformation is linear, so the jacobian term won't screw this factorization up when we actually pass to the joint pdf of $(A, \bar X)$.

Answer 2

Hint:

1. Show that $A' = (\bar X, A)$ can be writen as $A' = C X$ where $C$ is some square matrix. Deduce from that that $A'$ is jointly gaussian.

2. Recall that independence is equivalent to zero correlation for jointly gaussian variables. Show that $E(A_i \bar X) = 0$. Conclude that $A$ and $\bar X$ are independent.

Answer 3

Here is an alternative matrix approach. By replacing each $X_i$ by $(X_i - \mu)/\sigma$, we can assume wlog that the $X$'s are independent standard normal variables.

Pack $X_1,\ldots,X_n$ into an $n\times 1$ column vector $X$. The idea is to express $\sum X$ and $(X_1-\bar X,\ldots,X_n-\bar X)^T$ as matrix transformations of $X$. This is achieved by taking $U=(1,\ldots,1)$, a $1\times n$ row vector of ones (so that $UX=\sum X_i$), and defining the $n\times n$ matrix $$C:=I_n - \frac1n U^TU$$ (so that $CX$ has $i$th member $X_i-\bar X$). Check that $UX$ and $CX$ each have zero mean. Their covariance is $$E[(UX)(CX)^T]=E(UXX^TC^T)=UC^T=UC=U-\frac1nUU^TU.$$ But $UU^T=n$, so the covariance matrix is zero. Since $UX$ are $CX$ are jointly normal, this implies that they are independent, hence so are $\bar X$ and $S^2$, which are functions respectively of $UX$ and $CX$.

---

Why are $UX$ and $CX$ jointly normal? The product of a partitioned matrix with $X$: $$\begin{pmatrix}U\\\hdashline C\end{pmatrix}X $$ yields a $(n+1)\times 1$ multivariate normal vector; $UX$ is the first member, and $CX$ is the remaining subvector.

Answer 4

[This is from "Introduction to Mathematical Statistics" by Hogg, McKean, Craig]

Let ${ X _1, \ldots, X _n }$ be iid ${ N (\mu, \sigma ^2) }$ random variables. Consider the sample mean and sample variance

$${ \overline{X} = \frac{1}{n} \sum _{i = 1} ^{n} X _i, \quad S ^2 = \frac{1}{n-1} \sum _{i = 1} ^{n} (X _i - \overline{X}) ^2 . }$$

Thm: The sample mean ${ \overline{X} }$ and sample variance ${ S ^2 }$ are independent. Further the random variable

$${ T = \sqrt{n} \frac{\overline{X} - \mu}{S} }$$

is distributed as Student's ${ t - }$distribution with ${ (n-1) }$ degrees of freedom.

Pf: Consider the vector of ${ 1 }$s ${ \mathbb{1} = (1, \ldots, 1) ^T }$ and the transformation

$${ W = \begin{pmatrix} \overline{X} \\ X _1 - \overline{X} \\ \vdots \\ X _n - \overline{X} \end{pmatrix} = \begin{pmatrix} \frac{1}{n} \mathbb{1} ^{T} \\ I - \frac{1}{n} \mathbb{1} \mathbb{1} ^T \end{pmatrix} X . }$$

We see the random vector ${ W }$ is distributed as

$${ N _{n +1} \left( \begin{pmatrix} \frac{1}{n} \mathbb{1} ^{T} \\ I - \frac{1}{n} \mathbb{1} \mathbb{1} ^T \end{pmatrix} \mu \mathbb{1} , \, \begin{pmatrix} \frac{1}{n} \mathbb{1} ^{T} \\ I - \frac{1}{n} \mathbb{1} \mathbb{1} ^T \end{pmatrix} \sigma ^2 I \begin{pmatrix} \frac{1}{n} \mathbb{1} ^{T} \\ I - \frac{1}{n} \mathbb{1} \mathbb{1} ^T \end{pmatrix} ^T \right) }$$

that is

$${ N _{n + 1} \left( \begin{pmatrix} \mu \\ 0 _{n} \end{pmatrix} , \sigma ^2 \begin{pmatrix} \frac{1}{n} &0 _n ^T \\ 0 _n &I - \frac{1}{n} \mathbb{1} \mathbb{1} ^T \end{pmatrix} \right) . }$$

Especially ${ \overline{X} }$ and ${ (X _1 - \overline{X}, \ldots, X _n - \overline{X}) }$ are independent. Hence ${ \overline{X} }$ and ${ S ^2 = \frac{1}{n-1} \sum _{i = 1} ^{n} (X _i - \overline{X}) ^2 }$ are independent as needed.

Consider the normalised sums

$${ \sum _{i = 1} ^{n} \left( \frac{X _i - \mu}{\sigma} \right) ^2 \, \, \text{ and } \, \, \sum _{i = 1} ^{n} \left( \frac{X _i - \overline{X}}{\sigma} \right) ^2 . }$$

We have

$${ {\begin{align} \sum _{i = 1} ^{n} \left( \frac{X _i - \mu}{\sigma} \right) ^2 = &\, \sum _{i = 1} ^{n} \left( \frac{(X _i - \overline{X}) + (\overline{X} - \mu) }{\sigma} \right) ^2 \\ = &\, \sum _{i = 1} ^{n} \left( \frac{X _i - \overline{X}}{\sigma} \right) ^2 + \left( \frac{\overline{X} - \mu}{\sigma / \sqrt{n}} \right) ^2 . \end{align}} }$$

We see the last two terms are independent, hence taking MGFs

$${ (1 - 2t) ^{-n/2} = E\left[\exp(t \, (n-1) S ^2 / \sigma ^2) \right] (1 - 2t) ^{-1/2} }$$

that is

$${ \sum _{i = 1} ^{n} \left( \frac{X _i - \overline{X}}{\sigma} \right) ^2 \, \sim \, \chi ^2 (n-1) . }$$

The random variable

$${ T = \sqrt{n} \frac{\overline{X} - \mu}{S} = \frac{ (\overline{X} - \mu) / (\sigma / \sqrt{n}) }{\sqrt{ (n -1) S ^2 / (n-1) \sigma ^2}} }$$

is of the form

$${ \frac{N(0, 1) \text{ variable}}{\sqrt{(\chi ^2 (n-1) \text{ variable}) / (n-1)} } }$$

where the normal random variable and the ${ \chi ^2 (n-1) }$ random variable are independent. Hence it is distributed as Student's ${ t - }$distribution with ${ (n-1) }$ degrees of freedom, as needed. ${ \blacksquare }$