I read that it is because state-value function required knowledge of the model of the MDP and action-value function doesn't, but I don't understand why that is either. It seems to me that they are equivalence of each other.
$\begingroup$
$\endgroup$
asked Jan 2, 2017 at 9:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is only true when using temporal difference learning alone, i.e. Q-learning. In that setting you are learning the optimal state-action-value function Q* and then taking actions that maximize Q*. If instead, you learned V*, you know the real value of the state that you are in if you followed an optimal policy but that doesn't help you make a decision which action to choose (because V is not a function of a). If you knew the model, you could see what reward you would get for each action and what state you would end up in, thus effectively calculating Q* from V*.
