3
$\begingroup$

In model free learning, we use Action value over state value. I'm confused why is this the case?

I looked up on this and other forums and found out that we use action value over state value because In state value we need the transition probability which we don't have access to if the model does not exists in the first place

enter image description here

My question:

Even while using the Action value function for optimal policy ( equation below), to calculate the value of $q^*(s,a)$ we still need the transition probability. Therefore, even here how can we use action value when we don't have access to transition probability.

enter image description here

enter image description here

I know I'm missing something fundamental here but just cannot seem to find what it is. Any help is highly appreicated.

$\endgroup$
3
$\begingroup$

You have correctly quoted a relevant Bellman equation for action values. The state transition function is required there. In that context, the reward function $R(s,a,s')$ is also part of the model, so required to evaluate the expression.

Model-free methods in value-based Reinforcement Learning (RL) are based on the Bellman equation, but do not use it directly. For instance, in basic (single step tabular) Q learning, the value update step is:

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha(r_{t+1} + \gamma\text{max}_{a'}Q(s_{t+1}, a_{t+1}) - Q(s_t,a_t))$$

There are no mentions of $P(s'|s,a)$ or $R(s,a,s')$ in that update step. That is because in order to establish the values of $s_{t+1}$ and $r_{t+1}$ to use, these key values have been sampled from the environment.

This sampling is done by taking a step in a real or simulated environment. It is this environment that expresses the transition probabilities and reward function, the agent and learning algorithm do not have access to it or use it.

It is an important detail to note that sampling is occurring. In stochastic environments, it means that the agent needs to experience the same state and action multiple times, so that it can observe the multiple different ways that the environment can respond, and use the corresponding $s_{t+1}$ and $r_{t+1}$ values to perform updates to the action value estimate.

$\endgroup$
2
  • $\begingroup$ I had the same question. Can we not be sampling from an environment with a state-value-function (V)? I'm still a little unsure as to why Q over V is required for model-free. $\endgroup$ Jun 26 at 20:32
  • 1
    $\begingroup$ @alwayscurious Yours is a different question. In short you can learn the state value function for a given policy using a model-free approach. However, you cannot use that learned state value function to make action choices, because you do not have any way to predict which state will occur next depending on the action. This is a problem for value-based methods such as Monte Carlo control, SARSA and Q-learning which must all use action values. It is not a problem when combined with policy gradient as in Actor-Critic methods, because they provide a separate controllable policy. $\endgroup$ Jun 26 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.