In model free learning, we use Action value over state value. I'm confused why is this the case?
I looked up on this and other forums and found out that we use action value over state value because In state value we need the transition probability which we don't have access to if the model does not exists in the first place
Even while using the Action value function for optimal policy ( equation below), to calculate the value of $q^*(s,a)$ we still need the transition probability. Therefore, even here how can we use action value when we don't have access to transition probability.
I know I'm missing something fundamental here but just cannot seem to find what it is. Any help is highly appreicated.