Martingale formulation of Bellman's Optimality Principle

Question

From David Williams' Probability with Martingales

---

Related question: Deducing an optimal gambling strategy (using martingales).

---

What I tried:

For no 2, if $\ln Z_n - n \alpha$ is a supermartingale, then for $m < n$,

$$E[\ln Z_n - n \alpha | \mathscr F_m] \le \ln Z_m - m \alpha$$

$$ \to E[\ln Z_N - N \alpha | \mathscr F_0] \le \ln Z_0 - 0 \alpha$$

$$ \to E[\ln Z_N - N \alpha] \le \ln Z_0$$

$$ \to E[\ln \frac{Z_N}{Z_0}] \le N \alpha$$

Is that right?

---

For no 1, for $m < n$,

$$E[\ln Z_n - n \alpha | \mathscr F_m] = E[\ln Z_n | \mathscr F_m] - n \alpha$$

$$ = E[\ln \frac{Z_n}{Z_0} | \mathscr F_m] - n \alpha + \ln Z_0$$

$$ = E[\ln \frac{Z_n}{Z_{n-1}} | \mathscr F_m] + ... + E[\ln \frac{Z_{m+1}}{Z_{m}} | \mathscr F_m]$$

$$+ \ln \frac{Z_m}{Z_{m-1}} + ... + \ln \frac{Z_{1}}{Z_{0}} - n \alpha + \ln Z_0$$

Note that

$$Z_n - Z_{n-1} \le |Z_n - Z_{n-1}| \le C_n |\epsilon_n - \epsilon_{n-1}| \le 2C_n \le 2Z_{n-1}$$

Also, I think

$$\ln \frac{Z_n}{Z_{n-1}} \le |Z_n - Z_{n-1}|$$

Hence we have

$$E[\ln \frac{Z_n}{Z_{n-1}} | \mathscr F_m] + ... + E[\ln \frac{Z_{m+1}}{Z_{m}} | \mathscr F_m]$$

$$+ \ln Z_m - n \alpha$$

$$\le E[2Z_{n-1} | \mathscr F_m] + ... + E[2Z_{m} | \mathscr F_m]$$

$$+ \ln Z_m - n \alpha$$

$$= 2(E[Z_{n-1} | \mathscr F_m] + ... + E[Z_{m} | \mathscr F_m])$$

$$+ \ln Z_m - n \alpha$$

One thing to do would be to show that

$$2E[Z_{n-1} | \mathscr F_m] + ... + 2E[Z_{m} | \mathscr F_m] \le (n-m) \alpha$$

possibly by showing that $2E[Z_{\{\cdot\}} | \mathscr F_m] \le \alpha$

How would I do that?

Something else:

$$E[\ln \frac{Z_n}{Z_m} | \mathscr F_m] + \ln Z_m - n \alpha$$

$$ \le E[|Z_n - Z_m| | \mathscr F_m] + \ln Z_m - n \alpha$$

$$ \le E[2Z_m | \mathscr F_m] + \ln Z_m - n \alpha$$

$$ \le 2Z_m + \ln Z_m - n \alpha$$

Now $2Z_m \le (n-m) \alpha?$ We have

$$2Z_m = 2\sum_{k=1}^{m} C_k(\epsilon_k - \epsilon_{k-1})$$

$$\le 2\sum_{k=1}^{m} Z_{k-1}$$

I'm not quite sure how to show that

$$2\sum_{k=1}^{m} Z_{k-1} \le (n-m)\alpha$$

if that's even true.

How can/else can I approach this problem?

---

For no 3, no idea. Any hints? I think it has something to do with a stopping time. Might we have to use the ff lemma (red box):

?

---

Update 1 decade later: It's called freezing lemma I guess

$\mathbb{E}(\varphi(X, Y) | \mathcal{G}) = \mathbb{E}(\varphi(X, Y))$ What's the name for this proposition? Or alternatively what's happening?

Answer 1

Here is the solution to the questions.

Note that $Z_n = Z_{n-1} + \epsilon_n C_n$. We have $$\begin{align}\mathrm{E}\left[\log Z_n - \log Z_{n-1}\mid\mathcal{F}_{n-1}\right] &= \mathrm{E}\left[\log \left(1+\epsilon_n C_n/Z_{n-1}\right)\mid\mathcal{F}_{n-1}\right] \\ &= p\log\left(1+\frac{C_n}{Z_{n-1}}\right)+q\log\left(1-\frac{C_n}{Z_{n-1}}\right)\end{align}.$$ Consider $f(x) = p\log(1+x)+q\log(1-x)$. It reaches its maximum $\alpha$ at $x = p-q$. Therefore $$\mathrm{E}\left[{\log Z_n - \log Z_{n-1}}\mid{\mathcal{F}_{n-1}}\right] \le \alpha,$$ and so $\log Z_n - n\alpha$ is a supermartingale and it is a martingale if $C_n = (p-q)Z_{n-1}$ for all $n$. The best strategy hence is to follow $C_n = (p-q)Z_{n-1}$.