In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as

$$ J_\pi(\phi)=\mathbb E_{s_t\sim \mathcal D}\left[D_{KL}\left(\pi_\phi(\cdot|s_t)\Big\Vert {\exp(Q_\theta(s_t,\cdot)\over Z_\theta(s_t)}\right)\right] $$

Applying the reparameterization trick, let $a_t=f_\phi(\epsilon_t;s_t)$, then the objective could be rewritten as

$$ J_\pi(\phi)=\mathbb E_{s_t\sim \mathcal D, \epsilon \sim\mathcal N}[\log \pi_\phi(f_\phi(\epsilon_;s_t)|s_t)-Q_\theta(s_t,f_\phi(\epsilon_t;s_t))] $$

They compute the gradient of the above objective as follows

$$ \nabla_\phi J_\pi(\phi)=\nabla_\phi\log\pi_\phi(a_t|s_t)+(\nabla_{a_t}\log\pi_\phi(a_t|s_t)-\nabla_{a_t}Q(s_t,a_t))\nabla_\phi f_\phi(\epsilon_t;s_t) $$

The thing confuses me is the first term in the gradient, where does it come from? To my best knowledge, the second large term is already the gradient we need, why do they add the first term?

  • $\begingroup$ I just wanted to confirm that reinforcement learning is well within the scope of our site, as well as that of ai.stackexchange.com; despite the impression that may have been given by a now-deleted comment. (I've undeleted this question even though you've asked it here.) $\endgroup$ – Scortchi - Reinstate Monica Feb 13 '19 at 14:45
  • $\begingroup$ @Brale_ has answered this question here. $\endgroup$ – Maybe Mar 17 '19 at 13:12
  • $\begingroup$ Thanks for the update. $\endgroup$ – Scortchi - Reinstate Monica Mar 17 '19 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.