Reinforcement Learning - Why are actor critic methods biased? I understand that using a state-dependent baseline keeps the expected reward objective unbiased (whilst reducing the gradient variance) like in the equation below:
$$
\frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( ( \sum _ { t ^ { \prime } = t } ^ { T } \gamma ^ { t ^ { \prime } - t } r ( s _ { i , t ^ { \prime } } , a _ { i , t ^ { \prime } } ) ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) )
$$
However the actor-critic gradient (below) looks very similar i.e. it looks like it has the usual policy gradient with a state dependent baseline of
$$
\gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } )
$$
$$
\frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) )
$$
So my question is : why is the actor-critic algorithm biased given that it can be interpreted as a policy gradient objective with a state-dependent baseline much like the first equation.
 A: Key difference is in the estimation of the return. For policy gradient method (disregarding the baseline term which is only used to reduce variance) you have:  
$\sum_{t^{\prime} = t}^T \gamma^{t^{\prime} - t}r(s_{i,t^{\prime}}, a_{i,t^{\prime}})$ 
and for actor-critic method you have:  
$r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$
Policy gradient return estimate is unbiased because all rewards at all timesteps are sampled from the environment, so that gives us unbiased estimate.
Actor-critic return estimate is biased because $\hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$ term is biased. It is biased because it is an approximation of the expected return at state $s _ { i , t + 1 }$. This term is represented by an approximator, for example a neural network or a linear regression model. That approximator will usually be randomly initialized so it will not give a true estimation of the return, it will be biased towards some random value that was initialized with.
A: I'm going to add onto Brale's answer which I think is mostly right. The part that's slightly incorrect is to talk about the bias from initialization of the value neural network. I think it makes more sense to talk about the ideal value neural network (that minimizes the MSE objective).
Suppose we train the value network to convergence, obtaining a local minima. This process is itself unbiased (for the value objective).
However, at convergence, the value network imperfectly approximates the true value function given the policy. This error is bias in the value function.
So even if we trained our policy neural net using the converged value neural network, the (policy) objective is biased.
For concreteness, imagine freezing your policy net then "train your value net for infinite iterations" then make an update to the policy net. This update is biased since the value net is biased.
