1
$\begingroup$

I understand that using a state-dependent baseline keeps the expected reward objective unbiased (whilst reducing the gradient variance) like in the equation below:

$\frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( ( \sum _ { t ^ { \prime } = t } ^ { T } \gamma ^ { t ^ { \prime } - t } r ( s _ { i , t ^ { \prime } } , a _ { i , t ^ { \prime } } ) ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) )$

However the actor-critic gradient (below) looks very similar i.e. it looks like it has the usual policy gradient with a state dependent baseline of $\gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } )$

$\frac { 1 } { N } \sum _ { i = 1 } ^ { N } \sum _ { t = 1 } ^ { T } \nabla _ { \theta } \log \pi _ { \theta } ( a _ { i , t } | s _ { i , t } ) ( r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 } ) - \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t } ) )$

So my question is : why is the actor-critic algorithm biased given that it can be interpreted as a policy gradient objective with a state-dependent baseline much like the first equation.

$\endgroup$
1
$\begingroup$

Key difference is in the estimation of the return. For policy gradient method (disregarding the baseline term which is only used to reduce variance) you have:

$\sum_{t^{\prime} = t}^T \gamma^{t^{\prime} - t}r(s_{i,t^{\prime}}, a_{i,t^{\prime}})$

and for actor-critic method you have:

$r ( s _ { i , t } , a _ { i , t } ) + \gamma \hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$

Policy gradient return estimate is unbiased because all rewards at all timesteps are sampled from the environment, so that gives us unbiased estimate.
Actor-critic return estimate is biased because $\hat { V } _ { \phi } ^ { \pi } ( s _ { i , t + 1 })$ term is biased. It is biased because it is an approximation of the expected return at state $s _ { i , t + 1 }$. This term is represented by an approximator, for example a neural network or a linear regression model. That approximator will usually be randomly initialized so it will not give a true estimation of the return, it will be biased towards some random value that was initialized with.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.