Profit maximization with Markov Chains

Question

Problem : An opera singer is due to perform a long series of concerts. Having a bad temper, they are liable to pull out each night with probability $1/2$ . Once this has happened they will not sing again until the promoter convinces them of the promoter’s high regard. This the promoter does by sending flowers every day until the singer returns. Flowers costing $x$ thousand pounds, $0 ≤ x ≤ 1$, bring about a reconciliation with probability $\sqrt{x}$ .The promoter stands to make $£750$ from each successful concert. How much should they spend on flowers?

I'd like to have solution verification or alternative approaches .

(Problem's from exercise 1.10.4 of Markov Chains by J.R. Norris)

---

Interpretation : I suppose the singer pulls out independent of the past given that they performed last night. Also I suppose $x$ is "how much they[the promoter] should spend on flowers", which is time-homogeneous. I suppose promoter wants to maximize their expected profit.

---

Plan : I'll use a 2 state MC to track whether the singer performs or not . Then I'll find the long-run proportion of time that singer performs : $v(x)$ . Finally $x$ will be the maximizer of $f(x) = 750v(x)-x(1-v(x))$ , I think $f(x)$ should approximate the expected profit .

If the MC turns out to be irreducible , then by Theorem 1.10.2 , $v(x)$ will almost surely be the inverse of expected return time of the state that the singer performs .

Theorem 1.10.2 Let $P$ be irreducible and let $\lambda$ be any distribution . If $(X_n)_{n\ge 0}$ is Markov$(\lambda,P)$ then $$ \mathbb{P}\left( \frac{V_i(n)}{n} \to \frac{1}{m_i} \text{ as } n \to \infty \right) = 1 $$ where $V_i(n) = \sum_{k=0}^{n-1} 1_{\{X_k = i\}}$ and $m_i$ is expected return time to state $i$ .

---

Attempt : Let $(X_n)_{n\ge 0}$ be a Markov chain such that the initial distribution is uniform and $ X_n = \left\{\begin{array}{cc} 1 & \text{if singer performs } \\ 0 & \text{else } \end{array}\right. $ . So transition probabilities are $$ \left\{\begin{array}{cc} p_{10} = 1/2 , & p_{11} = 1/2 \\ p_{01} = \sqrt{x} , & p_{00} = 1 - \sqrt{x} \end{array}\right. $$ with $x\in (0,1] $ . $(X_n)_{n\ge 0}$ is irreducible on state space $\{0,1\}$ . The expected return time to $1$ is $m_1 = 1 + \frac{1}{2}\frac{1}{\sqrt{x}} $ , so $v(x) = 1/m_1 $ .

When $x = 0 $ , $\{0\}$ becomes the absorption state , the expected profit is $\frac{1}{2} 750 \sum_{x=1}^{\infty} x\left(\frac{1}{2}\right)^x = \frac{1}{2} 750(2) = 750 $ .

Numerically I found the maximum of $f(x) , x\in (0,1]$ is around $500$ at $x=1$ .

So should I conclude that he should spend $x=0$ GBP on her ?

Answer 1

Your setup of the transition matrix and expected profit seems right, but maybe not the rest of the analysis.

We want the stationary distribution $(\pi_0, \pi_1)$ which, by definition, is the solution to the system $$ \pi_0 = \pi_0 p_{00} + \pi_1 p_{01}$$ $$ \pi_1 = \pi_0 p_{10} + \pi_1 p_{11} $$ s.t. $\pi_0 + \pi_1 = 1.$ When I solve this, I get $$ \pi_0 = \frac{1}{1+2\sqrt{x}}, \quad \pi_1 = \frac{2\sqrt{x}}{1+2\sqrt{x}}.$$ Then, recognizing that $\pi_0$ is the long-term proportion of time that is spent in the "not performing" state, the expected daily profit (in thousands of pounds) is $$ f(x) = 0.75\pi_1 - x\pi_0 = \frac{1.5\sqrt{x} - x}{1 + 2\sqrt{x}}.$$ Differentiating w.r.t. $x$ and setting to 0 gives $$x_{max} = 0.25, \quad f(x_{max}) = 0.25.$$ That is, the club owner should spend 250 pounds on flowers every day the singer is sitting out for an expected daily club profit of 250 pounds. Notice that the optimal result has $\pi_0 = \pi_1 = 0.5$. That is, the best option is to keep the singer happy enough that she shows up half of the time.

Answer 2

Since the probability of the singer cancelling each night is one-half, the expected value each time she is convinced to perform is twice the profits from a single night, since the mean number of nights she performs in a row is two.

$750₤\cdot\sum_{i=0}^{\infty}\frac{1}{2^{i}} = 1500₤$

Note that I am assuming that by "pound" you are referring to the contemporaneous currency of Great Britain; however, the solution should still hold true even if you actually meant it as a unit of weight for some unspecified material.

As long as the promoter can on average convince the singer to perform again for less than fifteen hundred pounds, he is still turning a profit.

Since the expected value of each performance is $1500$ pounds, the probability of performance is $\sqrt{x}$, and the cost of flowers is $1000₤\cdot{x}$ , we can guarantee additional performances by spending $1000$ pounds on flowers every time the singer gets uppity. This means that our expected net profit is $1500 - 1000 = 500$ pounds, which we produce over a mean of three days for an expected daily profit of $\frac{500}{3} = 166.67$ pounds.

Now that we have proven that the promoter can consistently turn a profit, we must turn our eyes towards maximizing his profit by considering the rate at which he earns.

Since the probability of the singer returning to work on any given day is $\sqrt{x}$, the mean number of days that the promoter must entice her florally is $\frac{1}{\sqrt{x}}$, the total cost in that time is $1000\sqrt{x}$ pounds, and the mean time per cycle is $2 + \frac{1}{\sqrt{x}}$ days, our mean profit in pounds per day is $\frac{\left(1500-1000\sqrt{x}\right)}{2+\frac{1}{\sqrt{x}}}$ , whose maximum on the interval $[0,1]$ occurs at $x = 0.25$ for an expected profit of $250$ pounds per day.

Therefore, to maximize profits, the promoter should expend exactly $250$ pounds per day on flowers.

Edit for process clarification:

We can generalize this solution to other problems in a similar format by thinking of this in cycles, where each cycle has two components: A number of nights $a$ where the promoter makes a profit $P_a$, and a number of nights $b$ where the promoter pays $P_b$ per night. The expected nightly profit is then the time-weighted mean of the two components of the cycle, $E\left(C\right)=\frac{\left(P_{a}-P_{b}\right)}{a+b}$

Since the probability of ending the loss portion of the cycle depends on the amount spent per night, letting $x$ equal the amount spent and the probability of ending the loss portion of the cycle equal $f(x)$, the mean $b$ is equal to $\frac{1}{f\left(x\right)}$ and $P_{b}=x\cdot b=\frac{x}{f\left(x\right)}$ so $E\left(C\right)=\frac{P_{a}-x\cdot b}{a+b}=\frac{P_{a}-\frac{x}{f\left(x\right)}}{a+\frac{1}{f\left(x\right)}}$

In the original question, we find $P_a$ is equal to the nightly pay of 750 times the mean number of nights, $a = 2$, $f\left(x\right)=\sqrt{x}$ (where $x$ is measured in thousands of pounds), $b = \frac{1}{f\left(x\right)}=\frac{1}{\sqrt{x}}$, $P_b = x\cdot b=\frac{x}{f\left(x\right)}=\frac{x}{\sqrt{x}}=\sqrt{x}$, so $E\left(x\right)=\frac{P_{a}-Pb}{a+b}=\frac{1.5-\sqrt{x}}{2+\frac{1}{\sqrt{x}}}$ where the expected value for the cycle as a function of $x$ has thousands of pounds per day as the unit for its y-coordinate.

Answer 3

Suppose the opera singer has $n$ performance days. Let $X_i = \begin{cases} 1 & \text{if performance occurs on day $i$}\\ 0 & \text{else} \end{cases}$ for $1 \leq i \leq n$.

Then the profit is $M = \left (\frac{3}{4} + x \right )\left (X_1 + \dots + X_n\right ) - nx$ (in thousands of pounds).

We have transition matrix $$P = \begin{bmatrix} 1 - \sqrt{x} & \sqrt{x}\\ \frac{1}{2} & \frac{1}{2} \end{bmatrix} = \begin{bmatrix} 1 & -2\sqrt{x}\\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0\\ 0 & \frac{1}{2}-\sqrt{x} \end{bmatrix} \begin{bmatrix} \frac{1}{1 + 2\sqrt{x}} & \frac{2\sqrt{x}}{1 + 2\sqrt{x}}\\ -\frac{1}{1 + 2\sqrt{x}} & \frac{1}{1 + 2\sqrt{x}} \end{bmatrix},$$ so \begin{align*} X_1 &= \begin{bmatrix} \frac{1}{2}\\ \frac{1}{2} \end{bmatrix}\\ \implies X_i = X_1^TP^{i-1} &= \begin{bmatrix} \frac{1}{2}\\ \frac{1}{2} \end{bmatrix}^T \begin{bmatrix} 1 & -2\sqrt{x}\\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0\\ 0 & \left (\frac{1}{2}-\sqrt{x}\right )^{i-1} \end{bmatrix} \begin{bmatrix} \frac{1}{1 + 2\sqrt{x}} & \frac{2\sqrt{x}}{1 + 2\sqrt{x}}\\ -\frac{1}{1 + 2\sqrt{x}} & \frac{1}{1 + 2\sqrt{x}} \end{bmatrix}\\ &= \begin{bmatrix} \frac{1 - \left (\frac{1}{2} - \sqrt{x} \right )^i}{1 + 2\sqrt{x}}\\ \frac{2\sqrt{x} + \left (\frac{1}{2} - \sqrt{x} \right )^i}{1 + 2\sqrt{x}} \end{bmatrix} \end{align*} Thus, \begin{align*} \mathbb{E}[X_i] &= \frac{2\sqrt{x} + \left (\frac{1}{2} - \sqrt{x} \right )^i}{1 + 2\sqrt{x}}\\ \implies E_n(x) = \mathbb{E}[M] &= \left ( \frac{3}{4} + x \right ) \left (\sum_{i = 1}^n \frac{2\sqrt{x} + \left (\frac{1}{2} - \sqrt{x} \right )^i}{1 + 2\sqrt{x}}\right ) - nx\\ &= \left ( \frac{3}{4} + x \right ) \left (\frac{(4nx + 1) + 2(n - 1)\sqrt{x} - 2\left (\frac{1}{2} - \sqrt{x}\right )^{n+1}}{\left (1 + 2\sqrt{x}\right )^2}\right ) - nx\\ &= \left ( \frac{3}{4} + x \right ) \left (\frac{1 - 2\sqrt{x} - 2\left (\frac{1}{2} - \sqrt{x}\right )^{n+1}}{\left (1 + 2\sqrt{x}\right )^2}\right ) + \frac{n\left (3\sqrt{x} - 2x \right )}{2 + 4\sqrt{x}} \end{align*} Thus as $n \to \infty$, the total profit is infinite as long as $3\sqrt{x} > 2x$, i.e. $x > 0$.

On the other hand, if we wanted to maximise the eventual daily profit then we go back to our distribution for $X_i$ and take $i \to \infty$ to get $X = \begin{bmatrix} \frac{1}{1 + 2\sqrt{x}}\\ \frac{2\sqrt{x}}{1 + 2\sqrt{x}} \end{bmatrix}$, so the expected profit made is $\left (\frac{3}{4} + x\right )\mathbb{E}[X] - x = \frac{3\sqrt{x} - 2x}{2 + 4\sqrt{x}}$, which is maximised when $x = \frac{1}{4}$.

Therefore, to maximise daily profit, the club should spend $250$ pounds.