Reading the book by Sutton & Barto on Reinforcement Learning, I found the following:

The optimal action- value function allows optimal actions to be selected without having to know anything about possible successor states and their values, that is, without having to know anything about the environment’s dynamics. Page 65

Why is the case that the state-action value function does not depend on the dynamics of the system?


1 Answer 1


Why is the case that the state-action value function does not depend on the dynamics of the system?

All value functions, including the action value fully depend on dynamics of the system. The statement in the book is not contradicting that fact.

The statement in the book could be re-phrased as follows:

  • The action value function maps a state and action to the expected future return in the following time steps.
  • The action value includes the impact of selecting a specific action, unlike the state value function.
  • It is therefore possible to select an action using an action value function without directly using knowledge of the environment dynamics.

The comparison to make here is with state value functions, which do need to reference state dynamics in order to select the next action.

This can be seen in the equations for optimal policy using optimal state value and action value functions:

$$\pi^*(s) = \text{argmax}_a \sum_{r,s'}p(r,s'|s,a)(r + \gamma v^*(s'))$$

$$\pi^*(s) = \text{argmax}_a q^*(s,a)$$

The model of the environment dynamics is represented in the above equations as $p(r,s'|s,a)$, the probability of observing reward $r$ and next state $s'$ when starting in state $s$ and taking action $a$.

Clearly the policy does not need to reference the dynamics when using the action value $q^*(s,a)$. This is why value-based model-free control methods generally use action values.

  • $\begingroup$ Thanks, it is more clear now. Just one comment, the optimum policy is function of state and action, right? i.e. the formula would be: \pi * (a | s) = ..... $\endgroup$
    – Ralphns
    Dec 9, 2020 at 14:11
  • 1
    $\begingroup$ @Ralphns: Tere are two variants of policy functions, deterministic policies $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ and stochastic policies $\pi(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A_t = a | S_t = s\}$. Stochastic policies are more general, but in many environments, there are one or more deterministic optimal policies. Because that is the most common case, that is what I used in the answer. $\endgroup$ Dec 9, 2020 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.