Reinforcement Learning - Why are actor critic methods biased?

Question

I understand that using a state-dependent baseline keeps the expected reward objective unbiased (whilst reducing the gradient variance) like in the equation below:

$$ \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( ( \sum _ { t ^ { \prime } = t } ^ { T } \gamma ^ { t ^ { \prime } - t } r ( s _ { i , t ^ { \prime } } , a _ { i , t ^ { \prime } } ) ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) ) $$

However the actor-critic gradient (below) looks very similar i.e. it looks like it has the usual policy gradient with a state dependent baseline of

$$ \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) $$ $$ \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) ) $$

So my question is : why is the actor-critic algorithm biased given that it can be interpreted as a policy gradient objective with a state-dependent baseline much like the first equation.

Answer 1

Key difference is in the estimation of the return. For policy gradient method (disregarding the baseline term which is only used to reduce variance) you have:

$\sum_{t^{\prime} = t}^T \gamma^{t^{\prime} - t}r(s_{i,t^{\prime}}, a_{i,t^{\prime}})$

and for actor-critic method you have:

$r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$

Policy gradient return estimate is unbiased because all rewards at all timesteps are sampled from the environment, so that gives us unbiased estimate.
Actor-critic return estimate is biased because $\hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$ term is biased. It is biased because it is an approximation of the expected return at state $s _ { i , t + 1 }$. This term is represented by an approximator, for example a neural network or a linear regression model. That approximator will usually be randomly initialized so it will not give a true estimation of the return, it will be biased towards some random value that was initialized with.

Answer 2

I'm going to add onto Brale's answer which I think is mostly right. The part that's slightly incorrect is to talk about the bias from initialization of the value neural network. I think it makes more sense to talk about the ideal value neural network (that minimizes the MSE objective).

Suppose we train the value network to convergence, obtaining a local minima. This process is itself unbiased (for the value objective).

However, at convergence, the value network imperfectly approximates the true value function given the policy. This error is bias in the value function.

So even if we trained our policy neural net using the converged value neural network, the (policy) objective is biased.

For concreteness, imagine freezing your policy net then "train your value net for infinite iterations" then make an update to the policy net. This update is biased since the value net is biased.

Answer 3

The correct answer is because the return and the log-policy derivative are not independent, therefore you cannot replace one of the two with an (even unbiased) estimator and have guarantees that the gradient is still unbiased.

We can summarize the gradient as E[XY] where X is the return (sum of rewards), Y is the derivative of the log-policy, and the expectation is over (s,a) samples.

What actor-critic do is to replace X (the return) with an estimate X' (the one-step return in your equation). If:

1. Both X and X' are independent from Y,
2. X' is an unbiased estimate of X,

then we could do E[XY] = E[X]E[Y] = E[X']E[Y] = E[X'Y]. That is, the new gradient would be unbiased. However, X and Y are not independent: the choice of action affects the gradient of the action probability and the return given that action.

Therefore, it doesn't matter how you learn your critic: actor-critic are not guaranteed to be unbiased (and often are biased). This is why Sutton's book says (Section 13.5):

"Note that the bias in the gradient estimate is not due to bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method."