How to choose the baseline for policy gradient I learnt that the baseline could be V or Q (could it be advantage?), but how to choose among these different options? What is the rationale behind making these choices? Thanks.
 A: I will assume that you are asking about how to choose a reasonable baseline for the advantage function $A(s,a) = q(s,a)-b$. Let's first consider the point of choosing a good baseline. In short, you can use any scalar, or in other words, anything that does not depend on the action. Contrary to what you said, you cannot use the q-values as the baseline: if you did, the agent would not learn anything in that case because  $A(s,a) = 0\ \ \forall a$.
Say that in state $s$, you have three possible actions with q-values $q(s,a_1) = 999, q(s,a_2)=1000, q(s,a_3)=1001$.
In order to know which action to use, what you really want is a notion of how much better the action is than the other actions in that state, which is the definition of the advantage function. A reasonable choice that comes to mind is to subtract a baseline that is some kind of an average, which in this case is $b=1000$. Thus, you would compute $A(s,a_1) = -1, A(s,a_2)=0, A(s,a_3)=1$.  However, in general, you also want to take the weighted average, i.e. weigh the options with the probability that you would take the action, which is precisely the value function $V(s)$, a very good baseline to use.
