How can I understand REINFORCE with baseline is not a actor-critic algorithm?

I read Sutton's RL book and I found that in page 333

Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated.

The pseudo code of REINFORCE-with-baseline is

And the pseudo code of actor-critic is

In the above pseudo code, how can I understand bootstrapping, and I think REINFORCE-with-baseline and actor-critic are similar and it is hard for beginners to tell apart.

The difference is in how (and when) the prediction error estimate $\delta$ is calculated.

In REINFORCE with baseline:

$\qquad \delta \leftarrow G - \hat{v}(S_t,\mathbf{w})\qquad$ ; after the episode is complete

In Actor-critic:

$\qquad \delta \leftarrow R +\gamma \hat{v}(S',\mathbf{w}) - \hat{v}(S,\mathbf{w})\qquad$ ; online

Bootstrapping in RL is when the learned estimate $\hat{v}$ from a successor state $S'$ is used to construct the update for a preceding state $S$. This kind of self-reference to the learned model so far allows for updates at every step, but at the expense of initial bias towards however the model was initialised. On balance, the faster updates can often lead to more efficient learning. However the bias can lead to instability.

In REINFORCE, the final return $G$ is used instead, which is the same value as you would use in Monte Carlo control. The value of $G$ is not a bootstrap estimate, it is a direct sample of the return seen when behaving with the current policy. As a result it is not biased, but you have to wait to the end of each episode before applying updates.

• I really appreciate your answer. You are an expert in RL. Did you receive a PhD degree at RL? – GoingMyWay Apr 17 '18 at 13:09
• @AlexanderYau: No, my degree was a long time ago in Physics, and just a BA. However, I have spent a lot of time in last few months studying RL, and one way that is good to learn for me is to answer questions here on Stack Exchange. So I do tend to find and answer quite a few of them. – Neil Slater Apr 17 '18 at 14:19
• Good, did you learn RL by read Sutton's book? I am learning RL these days, and I will ask questions on RL on this site. – GoingMyWay Apr 17 '18 at 14:48
• @AlexanderYau: Yes Sutton's book has been the main source of my learning. Also, David Silver's course lectures: www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html – Neil Slater Apr 17 '18 at 14:55

I would complement The answer given by @Neil Slater and say that you have to know that there's 2 ways of reducing the variance of MC Reinforce and these are :

• Substracting a baseline
• Approximating the expected return rather than estimating it in a MC fashion

Reinforce with baseline only uses the first method, while the Actor-critic is using the second.

The algorithm you showed here and called actor-critic in Sutton's book is actually an Advantage Actor Critic and is using both techniques for reducing the variance.