0
$\begingroup$

I have read this Confusion around Bellman (update) operator and yet I am not clear about the difference between the two equations $V^{\pi }(s)=R(s,\pi (s))+\gamma \sum _{s'}P(s'|s,\pi (s))V^{\pi }(s').\ $ from https://en.wikipedia.org/wiki/Bellman_equation and enter image description here

that I read from here https://www.cs.cmu.edu/~mgormley/courses/10601-s17/slides/lecture26-ri.pdf and many other places. All helps and clarifications are welcome and will be very helpful. Thank you.

Improve this question
$\endgroup$
6
  • $\begingroup$ Can you give more information about what your problem is? You show two different equations, but don't really ask a question, so it is not clear where you are stuck. At the very least it would help if you explain which of the terms you understand or don't understand in each equation. Otherwise someone answering has to explain all the terms, which is unnecessary extra work if all you need to understand is difference between $\pi(s)$ and $\pi(a|s)$ for instance $\endgroup$
    – Neil Slater
    Commented Sep 4, 2019 at 6:34
  • $\begingroup$ I think the problem is that both equations refer to the same term on the left hand side but it is not clear why the right hand side is equal as well... $\endgroup$
    – Fabian Werner
    Commented Sep 4, 2019 at 8:06
  • $\begingroup$ For the answer: The reason why in the second equation, there is a sum over the $R$ term is that people sometimes use different setups: Is $R$ actually a random variable or a deterministic reward? In the second equation they view that as a random variable and therefore, if you sum over it you get $E[R_t|S_t=s]$ which the people from the first equation just define / take for granted as $R(s, \pi(s))$... i.e. the second equation is more general and in the setup in which the first one is used, it reduces to the first one. $\endgroup$
    – Fabian Werner
    Commented Sep 4, 2019 at 10:59
  • $\begingroup$ You might also be interested in (shamelessly pointing to my own answer) this great answer ;-) stats.stackexchange.com/questions/243384/… $\endgroup$
    – Fabian Werner
    Commented Sep 4, 2019 at 11:00
  • $\begingroup$ @Neil Slater the two equations suppose to mean the same, but I don't seem to understand why the are different. The second equation look to me as the sum or expectation of the first one. $\endgroup$
    – yk3003018
    Commented Sep 4, 2019 at 14:11

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.