Why is the state-action value function required when the model is unknown?

Question

At page 117 of the book "Reinforcement Learning: An Introduction" by (Andrew Barto and Richard S. Sutton), it is stated

With a model (e.g. a full DP or MDP), state values (e.g. $V(s)$ are sufficient to determine a policy - just look ahead to the next step and choose the best combination of reward and state. Without a model, however, state values alone are not sufficient. One must explicitly estimate the value of each action for the values - e.g $Q(s, a)$ to be useful in suggesting a policy.

What is an example that makes this obvious? Why can't one just use a form of Bellman's equation in $V(s)$ to determine the best policy via e.g. value iteration, without a state model?

Answer 1

The $V(s)$'s are sufficient to determine a policy precisely because you have access to the model. In particular you have access to information about the transition structure of the MDP (you know how to look ahead). Without a model you might know you want to get to $s$, but you don't know what actions to take to get there. Q values eliminate the need to estimate the underlying model by just learning what actions are good to take in a state.

This is discussed in the survey by Szepesvari on page 14 where it is noted that knowing either $Q^*$ or $V^*$, $P$ and $r$ suffice for an RL agent to act optimally. Using their notation, $P$ and $r$ are essentially the model.