Actor-Critic advantage is always zero?

Question

I'm following the actor-critic lecture from Cal's reinforcement learning course, and I came across this slide (emphasis not mine):

With their critic (the value function) trained using

The problem I see with this is that $\hat{V}_{\phi}^{\pi}(s_{i,t})$ is fit to the sampled reward sums of the current policy, $\sum_{t'=t}^T r(s_{i, t'}, a_{i, t'})$, meaning:

$$ \begin{align*} \hat{A}^{\pi}(s_{i,t}, a_{i,t}) & = r(s_{i,t}, a_{i,t}) + \hat{V}_{\phi}^{\pi}(s_{i,t+1}) - \hat{V}_{\phi}^{\pi}(s_{i,t}) \\ & \approx r(s_{i,t}, a_{i,t}) + \sum_{t'=t+1}^T r(s_{i, t'}, a_{i, t'}) -\sum_{t'=t}^T r(s_{i, t'}, a_{i, t'}) \\ & = 0 \end{align*} $$

The only way I can see this working is if $\hat{V}$ is not fit very well to the training data. If it is fit well, the advantages should all be 0, meaning the entire gradient is 0.

Since the lecture does not contain a detailed algorithm for step 2 (is it regularized to avoid overfitting to this one epoch--avoiding this problem?), I'm not sure how the critic is trained in a way that doesn't make the gradient always zero.

Answer 1

Your approximation of $\hat V^\pi_\phi(s_{i,t}) \approx \sum_{t' = t}^T r(s_{i,t'}, a_{i,t'})$ is way too crude to produce any meaningful interpretation.

Even if the value estimator is perfectly trained, it can only ever estimate the expected reward of the future trajectory -- not the exact reward from a particular rollout, since the policy is stochastic. So even in that case the advantage is not 0.

Furthermore, the policy is always being updated and changed, so it is unlikely that the value function would ever fit perfectly to the data until both the policy and the value function converge to some optimum.