Timeline for Proof of Bellman Optimality Equation
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 23, 2023 at 18:50 | comment | added | piero | @JieShi Moreover, $q_p(s,a)$ is not even defined. | |
| Sep 23, 2023 at 6:47 | comment | added | piero | @JieShi I don't get why the $p$ you defined is an optimal policy. | |
| Oct 25, 2019 at 18:43 | comment | added | Jie Shi | To endbegin: Unfortunately, we do not have an inverse version of this relationship. But we do have the following one: $V_{\pi}(s)=Q_{\pi}(s,a)-A(s,a)$ where $A(s,a)$ is called advantage function. | |
| Sep 16, 2019 at 23:41 | comment | added | endbegin | Do you have a similar derivation as (1) showing how $q_{\pi}(s,a)$ can be expressed as $v_{\pi}(s)$? | |
| Apr 19, 2019 at 4:35 | comment | added | Jie Shi | To your first concern, $p(\cdot)$ denotes probability distribution. Policy is, intrinsically speaking, distribution of action condition on state. In this sense, $p$ and $\pi$ can be used interchangeably. Hope you now understand how Eq. 1 comes. To your second concern, What you have written is equivalent to the result that we want to prove. | |
| Jan 21, 2019 at 16:39 | comment | added | hardhu | I am not sure if Jie Shi answer is an answer at all: first of all, it is not clear at all what is $p$ in the derivation of Eq. 1. I am guessing that you are referring to this relation between value and state-value functions for policies: $$v_{\pi}(s) = \sum_{a \in \mathcal{A}(s)} \pi(a|s) q_{\pi}(s,a)$$ But then if you define your deterministic policy $p$ as you have done, what you get by applying the previous relation to it is simply $$v_{p}(s) = q_{p}(s,a_s)$$ where $a_s \in \text{argmax}_{a \in \mathcal{A}(s)} q_*(s,a)$ | |
| Oct 4, 2018 at 22:20 | review | Late answers | |||
| Oct 4, 2018 at 22:50 | |||||
| Oct 4, 2018 at 22:01 | review | First posts | |||
| Oct 4, 2018 at 23:06 | |||||
| Oct 4, 2018 at 22:01 | history | answered | Jie Shi | CC BY-SA 4.0 |