2
$\begingroup$

This paper introduces Stochastic Gradient Meta RL (SGMLR). My question is specifically about the computation. One needs to calculate the Hessian, $\nabla^2_\theta J_i(\theta)$ which is given by the following equations: $$\nabla^2J_i(\theta)=\mathbb{E}_{\tau\sim q_i(\cdot,\theta)}[u_i(\tau,\theta)]$$ where $$u_i(\tau,\theta)=\nabla_\theta v_i\nabla_\theta \log q_i(\tau,\theta)^T+\nabla_\theta^2v_i(\tau,\theta)$$ where $$v_i(\tau,\theta)=\sum_{h=0}^H\log\pi_i(a_h|s_h)\mathcal{R}_i^h(\tau)$$

My question is about the second equation, specifically the term $\nabla_\theta \log q_i(\tau,\theta)$, where $q_i(\tau,\theta)$ is the probability of trajectory $\tau=(s_0,a_0,s_1,a_1,...,s_H,a_H)$ if the policy has parameters $\theta$. More precisely, I don't know how to calculate/estimate that term, and unfortunately the paper does not address that.

I was looking at their code here, but I can't seem to find where they calculate that term or what they do with that term. I'd appreciate any help.

$\endgroup$

1 Answer 1

1
$\begingroup$

The probability of the trajectory should be $q(\tau, \theta) = Pi_{i=0}^H \pi(s_i, a_i | \theta)$. That’s at least how we calculate the importance sampling correction factor in off-policy learning.

I’m not familiar with using $q(., \theta)$ as notation for that though; I’m more familiar with $q$ being used to denote the function that estimates the action values. I would check out [Sutton & Barto as a reference for this (pdf free available online).

$\endgroup$
1
  • $\begingroup$ Yes, but in practice one doesn't have access to $P$. The authors do indicate that probability is given by: $q_i(\tau,\theta)=\prod_{h=0}^H\pi_i\prod_{h=0}^HP_i$. Do you know how to estimate that value in practice? $\endgroup$
    – Schach21
    Mar 22 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.