Why do we use Action-value function over State-Value function in Model Free reinforcement learning

Question

In model free learning, we use Action value over state value. I'm confused why is this the case?

I looked up on this and other forums and found out that we use action value over state value because In state value we need the transition probability which we don't have access to if the model does not exists in the first place

My question:

Even while using the Action value function for optimal policy ( equation below), to calculate the value of $q^*(s,a)$ we still need the transition probability. Therefore, even here how can we use action value when we don't have access to transition probability.

I know I'm missing something fundamental here but just cannot seem to find what it is. Any help is highly appreicated.

Answer 1

You have correctly quoted a relevant Bellman equation for action values. The state transition function is required there. In that context, the reward function $R(s,a,s')$ is also part of the model, so required to evaluate the expression.

Model-free methods in value-based Reinforcement Learning (RL) are based on the Bellman equation, but do not use it directly. For instance, in basic (single step tabular) Q learning, the value update step is:

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha(r_{t+1} + \gamma\text{max}_{a'}Q(s_{t+1}, a_{t+1}) - Q(s_t,a_t))$$

There are no mentions of $P(s'|s,a)$ or $R(s,a,s')$ in that update step. That is because in order to establish the values of $s_{t+1}$ and $r_{t+1}$ to use, these key values have been sampled from the environment.

This sampling is done by taking a step in a real or simulated environment. It is this environment that expresses the transition probabilities and reward function, the agent and learning algorithm do not have access to it or use it.

It is an important detail to note that sampling is occurring. In stochastic environments, it means that the agent needs to experience the same state and action multiple times, so that it can observe the multiple different ways that the environment can respond, and use the corresponding $s_{t+1}$ and $r_{t+1}$ values to perform updates to the action value estimate.