1
$\begingroup$

By substituting the optimal value function into the Bellman equation, we get the Bellman equation for $v_{\star}$

$$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\limits_{s'} \sum_r p(s', r | s, a)[r + \gamma v_{\star}(s')]$$

From the above equation, how can we obtain the this one?:

$$ v_{\star}(s) = \max\limits_a \sum\limits_{s'} \sum\limits_r p(s', r|s,a) [r + \gamma v_{\star}(s')])$$

$\endgroup$

1 Answer 1

1
$\begingroup$

Maybe you should provide some context. The second equation can only be derived from the first one, if the policy is deterministic, so for a poker game this would not hold.

The second equation is then obtained from the first by the policy

\begin{equation} \pi_{\star}(a | s) = \begin{cases} 1 & \text{if a = }\text{a}_{\star},\\ 0 & \text{otherwise}\\ \end{cases}\end{equation} Where $a_{\star}$ is the best action that can be taken in $s$, hence $a_{\star} = \text{argmax}_a q(s,a)$.

Then \begin{equation} v_{\star}(s) = \sum_{a} \pi_{\star}(a | s) \sum_{s',r}p(s', r|s,a)[r + \gamma v_{\star}(s')] = \sum_{a} \pi_{\star}(a | s)q_{\star}(s, a) = q_{\star}(s, a_{\star}) = \max\limits_a q_{\star}(s, a) = \max\limits_a \sum_{s',r}p(s', r|s,a_{\star})[r + \gamma v_{\ast}(s')] \end{equation}

I used the equality $\sum_{s',r}p(s', r|s,a)[r + \gamma v_{\star}(s')] = q_{\star}(s, a)$ which is given by Sutton's "Reinforcement Learning: An Introduction" (second edition) at equation (3.20) combined with (3.19) in straight-forward fashion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.