9
$\begingroup$

I read Sutton's RL book and I found that in page 333

Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated.

The pseudo code of REINFORCE-with-baseline is enter image description here

And the pseudo code of actor-critic is

enter image description here

In the above pseudo code, how can I understand bootstrapping, and I think REINFORCE-with-baseline and actor-critic are similar and it is hard for beginners to tell apart.

$\endgroup$

3 Answers 3

10
$\begingroup$

The difference is in how (and when) the prediction error estimate $\delta$ is calculated.

In REINFORCE with baseline:

$\qquad \delta \leftarrow G - \hat{v}(S_t,\mathbf{w})\qquad$ ; after the episode is complete

In Actor-critic:

$\qquad \delta \leftarrow R +\gamma \hat{v}(S',\mathbf{w}) - \hat{v}(S,\mathbf{w})\qquad$ ; online

Bootstrapping in RL is when the learned estimate $\hat{v}$ from a successor state $S'$ is used to construct the update for a preceding state $S$. This kind of self-reference to the learned model so far allows for updates at every step, but at the expense of initial bias towards however the model was initialised. On balance, the faster updates can often lead to more efficient learning. However the bias can lead to instability.

In REINFORCE, the final return $G$ is used instead, which is the same value as you would use in Monte Carlo control. The value of $G$ is not a bootstrap estimate, it is a direct sample of the return seen when behaving with the current policy. As a result it is not biased, but you have to wait to the end of each episode before applying updates.

$\endgroup$
7
  • $\begingroup$ I really appreciate your answer. You are an expert in RL. Did you receive a PhD degree at RL? $\endgroup$
    – GoingMyWay
    Apr 17, 2018 at 13:09
  • 2
    $\begingroup$ @AlexanderYau: No, my degree was a long time ago in Physics, and just a BA. However, I have spent a lot of time in last few months studying RL, and one way that is good to learn for me is to answer questions here on Stack Exchange. So I do tend to find and answer quite a few of them. $\endgroup$ Apr 17, 2018 at 14:19
  • 1
    $\begingroup$ Good, did you learn RL by read Sutton's book? I am learning RL these days, and I will ask questions on RL on this site. $\endgroup$
    – GoingMyWay
    Apr 17, 2018 at 14:48
  • 2
    $\begingroup$ @AlexanderYau: Yes Sutton's book has been the main source of my learning. Also, David Silver's course lectures: www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html $\endgroup$ Apr 17, 2018 at 14:55
  • 1
    $\begingroup$ I guess, you are talking about bias in critic ($\delta$) in your answer, whereas Sutton Barto' are talking about bias in gradient and more specficially in actor ("... actor would be biased ..."), but not at all about bias in critic. So, I guess this is the exact reason why both are not contradicting each other. Right? $\endgroup$
    – Mahesha999
    Mar 31, 2021 at 18:40
4
$\begingroup$

I would complement The answer given by @Neil Slater and say that you have to know that there's 2 ways of reducing the variance of MC Reinforce and these are :

  • Substracting a baseline
  • Approximating the expected return rather than estimating it in a MC fashion

Reinforce with baseline only uses the first method, while the Actor-critic is using the second.

The algorithm you showed here and called actor-critic in Sutton's book is actually an Advantage Actor Critic and is using both techniques for reducing the variance.

$\endgroup$
1
$\begingroup$

Four years late to this post. Still have something to add...

I think REINFORCE-with-baseline and actor-critic are similar and it is hard for beginners to tell apart.

Neil's answer is great. But I guess the explanation in Sutton Barto's book sheds great light on above quoted doubt.

(RLBook, pdf page 353, book page 331, section 13.5 Actor-Critic Methods)

In REINFORCE with baseline, the learned state-value function estimates the value of the only the first state of each state transition. This estimate sets a baseline for the subsequent return, but is made prior to the transition’s action and thus cannot be used to assess that action. In actor–critic methods, on the other hand, the state-value function is applied also to the second state of the transition. The estimated value of the second state, when discounted and added to the reward, constitutes the one-step return, $G_{t:t+1}$, which is a useful estimate of the actual return and thus is a way of assessing the action. As we have seen in the TD learning of value functions throughout this book, the one-step return is often superior to the actual return in terms of its variance and computational congeniality, even though it introduces bias. We also know how we can flexibly modulate the extent of the bias with n-step returns and eligibility traces. When the state-value function is used to assess actions in this way it is called a critic, and the overall policy-gradient method is termed an actor–critic method. Note that the bias in the gradient estimate is not due to bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method.

To make it more intuitive, lets look at the update rools in both. More precisely, note that REINFORCE with baseline uses $G_t=\sum_{k=t+1}^T\gamma^{k-t-1}R_k$ and actor-critic uses $G_{t:t+1}=R_{t+1}+\gamma \hat{v}(S_{t+1},\bf{w})$:

enter image description here

enter image description here

enter image description here

PS1:

I didnt find your quoted text in Sutton Barto's book:

Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated.

$\endgroup$
1
  • $\begingroup$ The quoted text in my question was from a previous draft of the RLBook. $\endgroup$
    – GoingMyWay
    Apr 1, 2021 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.