How to compute the probability of trajectories term in Stochastic Gradient Meta Reinforcement Learning

Question

This paper introduces Stochastic Gradient Meta RL (SGMLR). My question is specifically about the computation. One needs to calculate the Hessian, $\nabla^2_\theta J_i(\theta)$ which is given by the following equations: $$\nabla^2J_i(\theta)=\mathbb{E}_{\tau\sim q_i(\cdot,\theta)}[u_i(\tau,\theta)]$$ where $$u_i(\tau,\theta)=\nabla_\theta v_i\nabla_\theta \log q_i(\tau,\theta)^T+\nabla_\theta^2v_i(\tau,\theta)$$ where $$v_i(\tau,\theta)=\sum_{h=0}^H\log\pi_i(a_h|s_h)\mathcal{R}_i^h(\tau)$$

My question is about the second equation, specifically the term $\nabla_\theta \log q_i(\tau,\theta)$, where $q_i(\tau,\theta)$ is the probability of trajectory $\tau=(s_0,a_0,s_1,a_1,...,s_H,a_H)$ if the policy has parameters $\theta$. More precisely, I don't know how to calculate/estimate that term, and unfortunately the paper does not address that.

I was looking at their code here, but I can't seem to find where they calculate that term or what they do with that term. I'd appreciate any help.

Answer 1

The probability of the trajectory should be $q(\tau, \theta) = Pi_{i=0}^H \pi(s_i, a_i | \theta)$. That’s at least how we calculate the importance sampling correction factor in off-policy learning.

I’m not familiar with using $q(., \theta)$ as notation for that though; I’m more familiar with $q$ being used to denote the function that estimates the action values. I would check out [Sutton & Barto as a reference for this (pdf free available online).