# Reinforcement Learning - Why are actor critic methods biased?

I understand that using a state-dependent baseline keeps the expected reward objective unbiased (whilst reducing the gradient variance) like in the equation below:

$$\frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( ( \sum _ { t ^ { \prime } = t } ^ { T } \gamma ^ { t ^ { \prime } - t } r ( s _ { i , t ^ { \prime } } , a _ { i , t ^ { \prime } } ) ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) )$$

However the actor-critic gradient (below) looks very similar i.e. it looks like it has the usual policy gradient with a state dependent baseline of $$\gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } )$$

$$\frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) )$$

So my question is : why is the actor-critic algorithm biased given that it can be interpreted as a policy gradient objective with a state-dependent baseline much like the first equation.

## 1 Answer

Key difference is in the estimation of the return. For policy gradient method (disregarding the baseline term which is only used to reduce variance) you have:

$$\sum_{t^{\prime} = t}^T \gamma^{t^{\prime} - t}r(s_{i,t^{\prime}}, a_{i,t^{\prime}})$$

and for actor-critic method you have:

$$r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$$

Policy gradient return estimate is unbiased because all rewards at all timesteps are sampled from the environment, so that gives us unbiased estimate.
Actor-critic return estimate is biased because $$\hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$$ term is biased. It is biased because it is an approximation of the expected return at state $$s _ { i , t + 1 }$$. This term is represented by an approximator, for example a neural network or a linear regression model. That approximator will usually be randomly initialized so it will not give a true estimation of the return, it will be biased towards some random value that was initialized with.