What's the relation between Generalized Policy Iteration (GPI), Actor-Critic, and Q-learning methods?

It seems to me that Generalized Policy Iteration (GPI) and Actor-Critic are the same, and Q-learning methods are a separate family of algorithms. I think both GPI and Actor-Critic describe the iterative process of policy evaluation (critic) and policy improvement (actor), while Q-learning is only bootstrapping using the Bellman optimality equation.

To elaborate on my understanding: Policy evaluation (critic) is done via Monte-Carlo or temporal difference methods, including function approximation if necessary, and policy improvement (actor) can be done by being greedy (tabular case) or using the policy gradient theorem (large state space). Q-learning is not doing any of these. It's just trying to estimate the $$Q^\ast$$ using the Bellman optimality equation by iteratively fitting the Bellman optimality equation for Q value.

I would appreciate it if anyone can confirm whether my understanding is correct or give a more systematic/precise summary of the taxonomy of online RL algorithms.

• Could you explain more about why Q-learning "still falls under GPI framework"? I thought Q-learning is looking for $Q^\ast$ directly, while "evaluation and improvement" should refer to $Q$. Also, if you count Q-learning as GPI, why is it not actor-critic then? (It can be parameterized.) Commented Jul 7 at 9:46
• From my understanding, Q-learning is constantly improving its estimate for $Q^\ast$, and $Q^\ast$ corresponds to the optimal policy $\pi^\ast$. Is $\pi^\ast$ the "implicit policy" you referred to? If so, it does not "improves" it, but improves the behavior policy. I think GPI (as shown in the picture you mentioned) must estimate and improve the same policy throughout. Commented Jul 7 at 23:44
• Here is my understanding, and I would appreciate confirmation/correction: In Q-learning (say, tabular case), a Q-table represents the estimate for $Q^\ast$. The "target policy" $\pi_t$ is greedy w.r.t. Q-table, and the "behavior policy" $\pi_b$ is $\epsilon$-greedy w.r.t. Q-table. The algorithm improves the Q-table via the Bellman optimality equation based on samples collected. Thus, both the target and the behavior policies are improved as they become closer to the optimal policy. So if it's GPI, the "evaluation" is for $\pi^\ast$, and "improvement" is for both $\pi_t$ and $\pi_b$. Commented Jul 8 at 0:59