Proof of the Central Limit Theorem using moment generating functions

Question

Below is a method of proving the Central Limit Theorem using moment generating functions.

Let $$X_{1},X_{2},...,X_{n}$$ be a sequence of i.i.d. random variables with expected value and variance $$E(X_{i}) = \mu < \infty, Var(X_{i})=\sigma ^{2}< \infty.$$

Now let

$$Z_{n}=\frac{\overline{X}-\mu }{\frac{\sigma }{\sqrt{n}}} = \frac{X_{1}+X_{2}+...+X_{n}-n\mu }{\sigma \sqrt{n}}.$$

We want to show that

$$\lim_{n \to \infty} M_{Z_{n}}(t)=e^{\frac{t^{2}}{2}}$$

where $M_{X}(t)$ is the moment generating function over some finite interval. In order to prove this, we can define a new random variable, $Y_{i}$, which is the normalized version of $X_{i}$. Thus,

$$Y_{i}=\frac{X_{i}-\mu }{\sigma }.$$
Then, we can say that $Y_{i}$ is i.i.d. with expected value and variance

$$E(X_{i}) =0, Var(X_{i})=1.$$

Using this information, we have $$Z_{n}=\frac{\overline{Y}-\mu }{\frac{\sigma }{\sqrt{n}}} = \frac{Y_{1}+Y_{2}+...+Y_{n} }{\sqrt{n}}.$$

Finding the moment generating function gives

$$M_{Z_{n}}(t)=E[e^{t\frac{Y_{1}+Y_{2}+...+Y_{n} }{\sqrt{n}}}] =E[e^{t\frac{Y_{1}}{\sqrt{n}}}]\cdot E[e^{t\frac{Y_{2}}{\sqrt{n}}}]\cdot ...\cdot E[e^{t\frac{Y_{n}}{\sqrt{n}}}]= M_{Y_{1}}(\frac{t}{\sqrt{n}})^{n}.$$

Lastly,

$$\lim_{n \to \infty} M_{Z_{n}}(t)=\lim_{n \to \infty} M_{Y_{1}}(\frac{t}{\sqrt{n}})^{n} = e^{\frac{t^{2}}{2}}.$$

This concludes the proof. However, how does one show analytically that this final limit does indeed equal

$$e^{\frac{t^{2}}{2}}?$$

angryavian · Accepted Answer · 2019-12-29T20:56:46.963

11

By Taylor's theorem, $$M_{Y_1}(s) = E[\exp(sY_1)] = 1 + s E[Y_1] + \frac{s^2}{2} E[Y_1^2] + s^2 h(s) = 1 + \frac{s^2}{2} + s^2 h(s), \qquad \text{where $h(s) \to 0$ as $s \to 0$},$$ where the last step uses $E[Y_1]=0$ and $\text{Var}(Y_1) = 1$.

Thus $$M_{Y_1}(t/\sqrt{n})^n = \left(1 + \frac{t^2/2}{n} + \frac{t^2}{n} h(t^2/n)\right)^n \to e^{t^2/2}.$$ [The expression in parentheses is asymptotically equivalent to $1+\frac{t^2/2}{n}$, so the last step follows by recalling $(1+\frac{x}{n})^n \to e^x$.]

Response to comment:

For $x$ near $0$, we have $\log(1+x) = x(1+g(x))$ where $g(x) \to 0$ as $x \to 0$.

\begin{align} &\lim_{n \to \infty} n \log M_{Y_1}(t/\sqrt{n}) \\ &= \lim_{n \to \infty} n(1-M_{Y_1}(t/\sqrt{n}))[1+g(1-M_{Y_1}(t/\sqrt{n}))] \\ &= \lim_{n \to \infty} \left\{n\left(\frac{t^2/2}{n} + \frac{t^2}{n}h(t^2/n)\right) \left[1+g\left(\frac{t^2/2}{n} + \frac{t^2}{n}h(t^2/n)\right)\right]\right\} \\ &= \lim_{n \to \infty} n\left(\frac{t^2/2}{n} + \frac{t^2}{n}h(t^2/n)\right) \cdot \lim_{n \to \infty} \left[1+g\left(\frac{t^2/2}{n} + \frac{t^2}{n}h(t^2/n)\right)\right] \\ &= \lim_{n \to \infty} n\left(\frac{t^2/2}{n} + \frac{t^2}{n}h(t^2/n)\right) \\ &= \frac{t^2}{2} + t^2 \lim_{n \to \infty}h(t^2/n) \\ &= \frac{t^2}{2}. \end{align}

edited Dec 29 '19 at 20:56

answered Apr 03 '18 at 05:22

angryavian

93,534

For your final step, you argue that it is justified to first take a limit within the parentheses, and then take a limit of the whole. Why would this be right? – Lukas Rollier Dec 29 '19 at 20:16
@Interestedstudent I have updated my answer with a more rigorous explanation – angryavian Dec 29 '19 at 20:57
@angryavian: Don't we have to show $\frac{t^2}{n} h(t^2/n)$ is $o(1/n)$? – Ashok Dec 16 '20 at 05:00
@Ashok No, all we need is that this term tends to zero as $n \to \infty$. This is true because $h(s) \to 0$ as $s \to 0$, based on the application of Taylor's theorem in the first sentence of my answer. – angryavian Dec 16 '20 at 14:41
@angryavian Sorry, how can that be? Is $\lim_{n\to\infty} (1+\frac{1}{n}+\frac{3}{n})^n = e$? – Ashok Dec 18 '20 at 05:49
1

@Ashok Sorry, my last comment was inaccurate. But doesn't $\lim_{s \to 0} h(s) = 0$ imply $\frac{t^2}{n} h(t^2/n) = o(1/n)$? The work below the horizontal line does use the fact that the term is $o(1/n)$. – angryavian Dec 18 '20 at 22:48
@angryavian: Yes, you are right! – Ashok Dec 19 '20 at 06:30
@angryavian: Do you have any good reference to the proof of the result "If $M_{X_n}(t)\to M_X(t)$, then $X_n\to^d X$? – Ashok Dec 24 '20 at 11:03
@Ashok The answers here and here have some discussion and references. – angryavian Dec 24 '20 at 19:22
@angryavian Thanks a lot! – Ashok Dec 25 '20 at 12:08
@angryavian Hello. I have a question about $h(s) \rightarrow 0$ as $s \rightarrow 0$. How can we be sure that some higher moments $E[Y^k]$ inside the term $h(s)$ are finite? What if some moment $E[Y^k]$ are infinite? Won't that cause $h(s)$ to explode? Please explain; thank you. – TrueWarrior09 Sep 02 '22 at 06:00
@angryavian Adding to my previous question: For example, I think a Pareto distribution can have finite mean and variance but infinite higher moments. In this case, how does the proof still work? – TrueWarrior09 Sep 02 '22 at 06:12
The proof that OP is using assumes the MGF exists, and cannot be applied in cases where it does not exist. However, there is a similar proof using characteristic functions instead. For characteristic functions, the existence of the first two moments implies the characteristic function is twice continuously differentiable, so the second-order Taylor approximation holds. @TrueWarrior09 – angryavian Sep 03 '22 at 02:13

Proof of the Central Limit Theorem using moment generating functions

1 Answers1

Linked