Policy gradient objective function

Question

Following Lecture 4 of the RL course at UC Berkeley, I'm wondering why they use this as the objective function:

Wouldn't a more accurate objective function be:

$$ \theta* = \arg\max_{\theta} E_{s_1 \sim p_1(s_1)} V_{\theta}(s_1) $$ $$V_{\theta}(s_{t, \; 1 \leq t \leq T}) = E_{a \sim \pi_{\theta}(a_t|s_{1:t})} \; r(s_t, a_t) + V_{\theta}(s_{t+1})$$ $$V_{\theta}(s_{t, \; t > T}) = 0$$

I say that because in the lecture's example, they multiply the entire path probability by the reward for that path, whereas the second objective function weighs earlier rewards only by the probability of the path-so-far. Intuitively, the second objective function makes more sense to me, so I don't see why the first objective function is used.

Answer 1

The objective function you have suggested makes an implcit assumption of boostrapping. Why just one step returns, why just two step returns, why n-step returns, why Monte-Carlo returns?

In expectation, these will all yield the same answers, but the rate of convergence will differ for different values of $n$, where $n-$step returns are being used as a target. In other words, the following will all be correct ways to write it in expectation, but variance reduction is a key aspect of Policy Gradient Techniques and Bootstrapping is employed for the same.

$$ \theta* = \arg\max_{\theta} E_{a \sim \pi_{\theta}(a_t|s_{1:t})} \; r(s_t, a_t) + V_{\theta}(s_{t+1}) $$

$$ \theta* = \arg\max_{\theta} E_{a_i \sim \pi_{\theta}(a_i|s_{1:i})} \; r(s_t, a_t) +r(s_{t+1}, a_{t+1})+\dots + r(s_{t+n}, a_{t+n})+V_{\theta}(s_{t+n+1}) $$

$$ \theta* = \arg\max_{\theta} E_{a_i \sim \pi_{\theta}(a_i|s_{1:i})} \; r(s_t, a_t) +r(s_{t+1}, a_{t+1})+\dots + r(s_{T}, a_{T}) $$

They are all the same in expectation, but different ones may be suited for different purposes.

• The first option may be useful for variance reduction
• The second option, which is an intermediate between TD(0) return and MC return is almost always the best. But hyperparameter tuning on $n$ is required
• MC (Monte Carlo) returns may be useful when the assumption of an MDP is a little strong and the actual underlying dynamics are slightly non-Markov