0

Given a finite irreducible Markov chain with transition matrix $P$ and stationary distribution $\pi$, how would you prove the following for any initial distribution vector $v$?

$$\lim_{n \to \infty} \frac{v\sum\limits_{k=1}^n P^k}{n} = \pi$$

Inside the limit is the expected number of times we visit each state over the first $n$ time steps.

This is a special case of the elementary renewal theorem (see here), but I'm hoping for a direct proof from the limit above.

If I assume the limit exists, I can prove that it satisfies $\pi = \pi P$, but how can I prove the limit exists?

A similar question was asked here, but existence of the limit was not shown.

Euclid
  • 3
  • It's not clear how much your question asks us to pretend we not know. May we assume the fixed-point equation $\mu=\mu P$ has only one probability vector solution? – kimchi lover Jun 13 '20 at 14:42
  • @kimchilover yes, you can use existence and uniqueness. I'm really just asking for a proof that the limit exists. – Euclid Jun 13 '20 at 14:48
  • This is proven rigorously in Theorem 54 of chapter 1 of Serfozo's Basics of Applied Stochastic Processes. I would suggest reading it carefully. The book can be found online at Yale's website: http://www.stat.yale.edu/~jtc5/251/readings/Basics%20of%20Applied%20Stochastic%20Processes_Serfozo.pdf – Math1000 Jun 14 '20 at 11:00
  • @Math1000 Hi, thanks for the reference, but although it looks similar, that result is actually different. The book does mention my question as Remark 78 on page 49, but it isn't proven. (It may be proven later on as a special case of the elementary renewal theorem, but that's not what i'm looking for) – Euclid Jun 14 '20 at 14:50
  • @user8675309 YES! that is exactly the same question. Thank you. – Euclid Jun 14 '20 at 14:52

1 Answers1

1

If $P$ is irreducible, and if $\pi=\pi P$ is a stationary distribution, then all components of $\pi$ are positive.

From this it follows that $P$ has at most one stationary distribution. For if $\alpha=\alpha P$ and $\beta=\beta P$ are distinct stationary distributions, then $\gamma=(\beta-m\alpha)/(1-m)$ is also a stationary distribution, where $m=\min_i \beta_i/\alpha_i$. But if $j$ is such that $m=\beta_j/\alpha_j$, then $\gamma_j = 0$, contradicting the previous result.

The set of probability vectors is compact, so the sequence $s_n=v\sum_{i=1}^nP^i / n$ has convergent subsequences. Let $\alpha$ be the limit of such a subsequence:$$ \alpha=\lim_{i\to\infty} s_{n_i}.$$ It is easy to check that $\alpha$ is stationary, since $$ \alpha P-\alpha =\lim_{i\to\infty} \frac{v (P^{n_i+1}-I)}{n_i}=0.$$ That is, any limit point of $s_n$ is stationary. But we saw $P$ has at most one stationary distribution. Since all the limit points of subsequences of $s_n$ are the same, the sequence $s_n$ itself converges to a stationary distribution.

kimchi lover
  • 24,981
  • We don't know that P is a-periodic, but existence and uniqueness doesn't require $P$ to be a-periodic, so your proof still works. Thank you! – Euclid Jun 14 '20 at 15:19
  • I guess another similar approach would be to prove the a-periodic case (which is simple because $P$ being a-periodic, means that $P^n$ approaches $\pi$). Then, for the case where $P$ has period $d$, we could break the sum into d different sums each of which is like the a-periodic case. Thanks again! – Euclid Jun 14 '20 at 15:27
  • Fixed typos. I meant irreducible but wrote aperiodic. – kimchi lover Jun 14 '20 at 15:43