I'm reading near-optimal representation learning for hierarchical reinforcement learning. At page 5, they obtain Equation 13 as follows

$$ \begin{align} J(\theta,s_t,\pi)&=\sum_{k=1}^c\gamma^{k-1}w_kD_{KL}(P_\pi(s_{t+k}|s_t)\Vert K_{\theta}(s_{t+k}|s_t,\pi))\tag {11}\\ &=B+\sum_{k=1}^c-\gamma^{k-1}w_k\mathbb E_{P_\pi(s_{t+k}|s_t)}[\log K_\theta(s_{t+k}|s_t,\pi)]\tag{12}\\ &=B+\sum_{k=1}^c-\gamma^{k-1}w_k\mathbb E_{P_\pi(s_{t+k}|s_t)}[\log E_\theta(s_{t+k},s_t,\pi)]+\gamma^{k-1}w_k\log \mathbb E_{\tilde s\sim\rho}[E_\theta(\tilde s,s_t,\pi)]\tag {13} \end{align} $$ where $E_\theta(s',s,\pi)=\exp(-D(f_\theta(s'),\varphi_\theta(s,\pi)))$ and $K_\theta(s'|s,\pi)\propto\rho(s')E_\theta(s',s,\pi)$. My question is where the last term comes from? By analyzing Equation 12, I get

$$ J(\theta,s_t,\pi)=B+\sum_{k=1}^c-\gamma^{k-1}w_k\mathbb E_{P_\pi(s_{t+k}|s_t)}[\log E_\theta(s_{t+k},s_t,\pi)]-\gamma^{k-1}w_k\mathbb E_{P_\pi(s_{t+k}|s_t)}[\log \rho(s')] $$

I cannot see the connection between the last term in the above equation and the last in Equation 13. Furthermore, it seems to me that the last term in the above equation is a constant w.r.t. $\theta$. So where is the last term in Equation 13 come from?


What I'm missing here is that $K_\theta(s'|s,\pi)$ is proportional to, not equal to, $\rho(s')E_\theta(s',s,\pi)$. Therefore, to express $K_\theta(s'|s,\pi)$ as a probability distribution, we have to renormalize it, which gives us $$ K_\theta(s'|s,a)={\rho(s')E_\theta(s',s,\pi)\over \int\rho(s')E_\theta(s',s,\pi)ds'} $$ Putting this back into Equation 12, we can get Equation 13.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.