Does state-Action Value function depends on environment's dynamics?

Question

Reading the book by Sutton & Barto on Reinforcement Learning, I found the following:

The optimal action- value function allows optimal actions to be selected without having to know anything about possible successor states and their values, that is, without having to know anything about the environment’s dynamics. Page 65

Why is the case that the state-action value function does not depend on the dynamics of the system?

Answer 1

Why is the case that the state-action value function does not depend on the dynamics of the system?

All value functions, including the action value fully depend on dynamics of the system. The statement in the book is not contradicting that fact.

The statement in the book could be re-phrased as follows:

• The action value function maps a state and action to the expected future return in the following time steps.
• The action value includes the impact of selecting a specific action, unlike the state value function.
• It is therefore possible to select an action using an action value function without directly using knowledge of the environment dynamics.

The comparison to make here is with state value functions, which do need to reference state dynamics in order to select the next action.

This can be seen in the equations for optimal policy using optimal state value and action value functions:

$$\pi^*(s) = \text{argmax}_a \sum_{r,s'}p(r,s'|s,a)(r + \gamma v^*(s'))$$

$$\pi^*(s) = \text{argmax}_a q^*(s,a)$$

The model of the environment dynamics is represented in the above equations as $p(r,s'|s,a)$, the probability of observing reward $r$ and next state $s'$ when starting in state $s$ and taking action $a$.

Clearly the policy does not need to reference the dynamics when using the action value $q^*(s,a)$. This is why value-based model-free control methods generally use action values.