0

I am studying Markov Rewaed Processes right now, and I wish to gain a deeper understanding of the Bellman equation's relationship with them.

I learned the Bellman equation in the following form:

$v = R + \gamma Pv$

Here, $R$ is the reward vector, $P$ the transition probability matrix, $\gamma$ is the discount factor and $v$ is the value we are trying to find/assign. The equation can be rearranged to

$v = (I - \gamma P)^{-1} R$

However, this rearrangement is only possible if (I - γP) is invertible. Is there a concise way to state the conditions under which this is the case?

chsgs
  • 1
  • Hi, welcome to Math SE. Please use MathJax to improve your question's formatting. Would "$1$ isn't an eigenvalue of $\gamma P$" suit your purposes? – J.G. Jan 30 '24 at 23:54
  • Hey, thanks! Just got to it, didn't figure Latex notation would be immediately supported haha. And yes, that does suit my purposes. I'm not sure how.to proceed from there, though. How do I, for an arbitrary $n \times n$ transition probability determine if it has an Eigenvalue of 1? – chsgs Jan 31 '24 at 00:22
  • https://math.stackexchange.com/questions/351142/why-markov-matrices-always-have-1-as-an-eigenvalue – chsgs Jan 31 '24 at 00:30
  • That doesn't help with the case $\gamma\notin{0,,1}$. – J.G. Jan 31 '24 at 07:48

0 Answers0