7

Consider the following one-player game: Initially, there is a pot of $\$0$. On each turn, the player may opt to end the game and collect the contents of the pot, or roll a single die. In the latter case, if the roll is $1$, the game is over and the player collects nothing. Otherwise, the amount of the die roll (from $\$2–\$6$) is added to the pot, and the player gets another move. The player continues rolling until they either get a $1$, and collect nothing, or opt to collect the pot. We would like to find the optimal strategy for this game, and the expected payoff under the optimal strategy.

A simple (but not quite correct) analysis goes like this: Any rational strategy must take the form “Stop if and only if the pot has value at least $\$S$.” If the pot contains $\$P$, then the expected gain from the next die roll is $\frac16(-P+2+3+4+5+6 )$ dollars. This is positive whenever $P\ge 20$, so the player should roll until the pot reaches $\$20$ or more, and then stop and collect it.

This is in fact an optimal strategy, but the analysis is not exactly correct, because it assumes, incorrectly, that the expected gain from a successful die roll is exactly $\$4$. But actually the expectation is a little bit more than this, because a successful die roll not only adds $\$4$ to the pot, it enables further die rolls that may add more. In fact strategies $S=20$ and $S=21$ yield identical expected payoffs of $\$8.141794893727$.

A more careful analysis follows: Let $X_S(P)$ be the expected amount won by following the stop-at-$S$ strategy when $\$P$ is already in the pot. Then $X_S$ satisfies the following unusual recurrence:

$$X_S(P) = \begin{cases} P & \text{if $P\ge S$} \\ \frac16\sum_{i=2}^6 X_S(P+i) & \text{otherwise} \end{cases} $$

We are interested in the behavior of $X_S(0)$, which we can calculate exactly using this recurrence. This is tedious, so I had the computer do it. $X_S(0)$ is maximized, as already mentioned, for $S=20$ and $S=21$.

My questions are:

  1. Is there an easier method to calculate $X_S(0)$ without the help of the computer?
  2. Is there an argument that shows that $X_{20}(0) = X_{21}(0)$?
MJD
  • 67,568
  • 43
  • 308
  • 617

1 Answers1

2

Let $X(P)$ denote the expected value of the game in its entirety, with \$$P$ in the pot. Thus $$X(P)=\max(P,\frac{1}{6}\left(0+X(P+2)+X(P+3)+X(P+4)+X(P+5)+X(P+6)\right)$$

Because $X(P)\ge P$, we have $X(P+2)\ge P+2$ (etc) and plug in and have $$X(P)\ge \frac{5P+20}{6}$$

Now for $P< 20$, we have $P< \frac{5P+20}{6}$ so we will not cash out until at least $P=20$. On the other hand, for all $k\in\mathbb{N}$, we have $X(P+k)\le X(P)+k$. Proof: Play two games in parallel, one with an extra \$k in the pot. If you bust, the extra money doesn't help, and if you cash out you've got an extra \$k.

Now, suppose $X(P)>P$ but $X(P-1)=P-1$. We have $$6(P-1)\ge X(P+1)+X(P+2)+X(P+3)+X(P+4)+X(P+5)$$ $$X(P+2)+X(P+3)+X(P+4)+X(P+5)+X(P+6)>6P$$ $$6P-6-X(P+1)>6P-X(P+6)$$ $$X(P+1)+5\ge X(P+6)>X(P+1)+6$$

This contradiction proves that once $X(P)=P$ then $X(P+k)=P+k$ for all $k\in \mathbb{N}$. Further, for some $P$ sufficiently large we must have $X(P)=P$ since otherwise the right strategy would be to play forever, which yields an expected value of \$0 almost surely. Set $P^\star$ to be the maximal value such that $X(P^\star)>P^\star$; from the above we have $P^\star\ge 19$.

We have $P^\star<20$ since $$6P^\star<X(P^\star+2)+X(P^\star+3)+X(P^\star+4)+X(P^\star+5)+X(P^\star+6)=5P^\star+20$$

Thus $P^\star=19$, and the optimal strategy is to keep rolling while we have 19 or less. To find the expected value of the original game, i.e. $X(0)$, we need to work backwards.

$X(20)=20, X(21)=21$, etc.

$X(19)=\frac{1}{6}(21+22+23+24+25)=\frac{115}{6}$

$X(18)=\frac{1}{6}(20+21+22+23+24)=\frac{55}{3}$

$X(17)=\frac{1}{6}(\frac{115}{6}+20+21+22+23)=\frac{631}{36}$

This is too many to calculate by hand, but I'm sure someone can write a few lines of code to figure it out.

vadim123
  • 83,937