2
$\begingroup$

Is this same as the Advantage Function?

$\endgroup$

1 Answer 1

3
$\begingroup$

A baseline function can be any function that doesn't affect the expected policy gradient update. Generally any function that does not directly depend on the current action choice or parametric policy function.

The idea for choosing a "good" baseline is to choose something that can correct for variance due to using the raw policy gradient for updates.

You actually have a lot of choices for the baseline function. Typical choices are $\bar{R}$, the average reward, or $V(s)$ the state value function (which can then lead to Actor-Critic methods, if you use it further to calculate a TD error for correcting the policy on each step).

Is this same as the Advantage Function?

The Advantage function can be treated as a baseline-already-included estimate, because it is $Q_{\pi}(s,a) - V_{\pi}(s)$. By using $A(s,a)$ as the gradient multiplier for policy gradient update steps, you are implictly using policy gradients with $V(s)$ as the baseline.

You might be able to use the Advantage function in some other way to establish a different baseline. However, you should not use the current policy to decide the action in $A(s,a)$ if you do so, because that is not independent of the policy gradient. You could use e.g. $\text{max}_a A(s,a)$ - I am not sure how well that would work in practice, but I think it would be a valid baseline function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.