I'm following the actor-critic lecture from Cal's reinforcement learning course, and I came across this slide (emphasis not mine):
With their critic (the value function) trained using
The problem I see with this is that $\hat{V}_{\phi}^{\pi}(s_{i,t})$ is fit to the sampled reward sums of the current policy, $\sum_{t'=t}^T r(s_{i, t'}, a_{i, t'})$, meaning:
$$ \begin{align*} \hat{A}^{\pi}(s_{i,t}, a_{i,t}) & = r(s_{i,t}, a_{i,t}) + \hat{V}_{\phi}^{\pi}(s_{i,t+1}) - \hat{V}_{\phi}^{\pi}(s_{i,t}) \\ & \approx r(s_{i,t}, a_{i,t}) + \sum_{t'=t+1}^T r(s_{i, t'}, a_{i, t'}) -\sum_{t'=t}^T r(s_{i, t'}, a_{i, t'}) \\ & = 0 \end{align*} $$
The only way I can see this working is if $\hat{V}$ is not fit very well to the training data. If it is fit well, the advantages should all be 0, meaning the entire gradient is 0.
Since the lecture does not contain a detailed algorithm for step 2 (is it regularized to avoid overfitting to this one epoch--avoiding this problem?), I'm not sure how the critic is trained in a way that doesn't make the gradient always zero.