How can I understand REINFORCE with baseline is not a actor-critic algorithm? I read Sutton's RL book and I found that in page 333

Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated.

The pseudo code of REINFORCE-with-baseline is

And the pseudo code of actor-critic is 

In the above pseudo code, how can I understand bootstrapping, and I think REINFORCE-with-baseline and actor-critic are similar and it is hard for beginners to tell apart.
 A: The difference is in how (and when) the prediction error estimate $\delta$ is calculated.
In REINFORCE with baseline:
$\qquad \delta \leftarrow G - \hat{v}(S_t,\mathbf{w})\qquad$ ; after the episode is complete
In Actor-critic:
$\qquad \delta \leftarrow R +\gamma \hat{v}(S',\mathbf{w}) - \hat{v}(S,\mathbf{w})\qquad$ ; online
Bootstrapping in RL is when the learned estimate $\hat{v}$ from a successor state $S'$ is used to construct the update for a preceding state $S$. This kind of self-reference to the learned model so far allows for updates at every step, but at the expense of initial bias towards however the model was initialised. On balance, the faster updates can often lead to more efficient learning. However the bias can lead to instability.
In REINFORCE, the final return $G$ is used instead, which is the same value as you would use in Monte Carlo control. The value of $G$ is not a bootstrap estimate, it is a direct sample of the return seen when behaving with the current policy. As a result it is not biased, but you have to wait to the end of each episode before applying updates.
A: I would complement The answer given by @Neil Slater and say that you have to know that there's 2 ways of reducing the variance of MC Reinforce and these are :


*

*Substracting a baseline

*Approximating the expected return rather than estimating it in a MC fashion


Reinforce with baseline only uses the first method, while the Actor-critic is using the second. 
The algorithm you showed here and called actor-critic in Sutton's book is actually an Advantage Actor Critic and is using both techniques for reducing the variance.
A: Four years late to this post. Still have something to add...

I think REINFORCE-with-baseline and actor-critic are similar and it is hard for beginners to tell apart.

Neil's answer is great. But I guess the explanation in Sutton Barto's book sheds great light on above quoted doubt.
(RLBook, pdf page 353, book page 331, section 13.5 Actor-Critic Methods)

In REINFORCE with baseline, the learned state-value function estimates the value of
the only the first state of each state transition. This estimate sets a baseline for the
subsequent return, but is made prior to the transition’s action and thus cannot be used
to assess that action. In actor–critic methods, on the other hand, the state-value function is applied also to the second state of the transition. The estimated value of the second state, when discounted and added to the reward, constitutes the one-step return, $G_{t:t+1}$, which is a useful estimate of the actual return and thus is a way of assessing the action.
As we have seen in the TD learning of value functions throughout this book, the one-step
return is often superior to the actual return in terms of its variance and computational
congeniality, even though it introduces bias. We also know how we can flexibly modulate
the extent of the bias with n-step returns and eligibility traces.
When the state-value function is used to assess actions in this way it is called a critic, and the overall policy-gradient method is termed an actor–critic method. Note that the bias in the gradient estimate is not due to bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method.

To make it more intuitive, lets look at the update rools in both. More precisely, note that REINFORCE with baseline uses $G_t=\sum_{k=t+1}^T\gamma^{k-t-1}R_k$ and actor-critic uses $G_{t:t+1}=R_{t+1}+\gamma \hat{v}(S_{t+1},\bf{w})$:



PS1:
I didnt find your quoted text in Sutton Barto's book:

Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated.

