1
$\begingroup$

I read the paper of DDPG

Many approaches in reinforcement learning make use of the recursive relationship known as the Bellman equation: $$ Q^{\pi}(s_t, a_t) = \mathbb{E}_{r_t, s_{t+1} \sim E} \left[ r(s_t, a_t) + \gamma \mathbb{E}_{a_{t + 1} \sim \pi} \left[ Q^{\pi}(s_{t+1}, a_{t + 1}) \right] \right] $$ If the target policy is deterministic we can describe it as a function $\mu: \mathcal{S} \leftarrow \mathcal{A}$ and avoid the inner expectation: $$ Q^{\mu}(s_t, a_t) = \mathbb{E}_{r_t, s_{t+1} \sim E} \left [ r(s_t, a_t) + \gamma Q^{\mu}(s_{t+1}, \mu(s_{t + 1})) \right] $$ The expectation depends only on the environment. This means that it is possible to learn $Q^{\mu}$ off-policy, using transitions which are generated from a different stochastic behavior policy $\beta$.

And I don't understand why The expectation depends only on the environment causes This means that it is possible to learn $Q^{\mu}$ off-policy, using transitions which are generated from a different stochastic behavior policy $\beta$?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.