3
$\begingroup$

When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only need the advantage function A(s, a) that describes the relative future reward for these actions, instead of the Q-function Q(s, a). Then is it possible to write Q-learning in such a way that it only involves the advantage function? By reading the paper on double DQN my impression is we need to compute both advantage function A(s, a) and value function V(s) explicitly instead of just the advantage function, but I don't see why that is the case.

$\endgroup$

2 Answers 2

1
$\begingroup$

When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only need the advantage function A(s, a) that describes the relative future reward for these actions, instead of the Q-function Q(s, a).

This is true for determining a current policy. However, this doesn't cover estimating the value function that you want to use from experience.

Then is it possible to write Q-learning in such a way that it only involves the advantage function?

You need an update mechanism for the value functions that can process experience and update the estimates for the value functions. Q learning is based on TD learning, which is derived from the Bellman equations for the value functions.

There is no simple Bellman equation for the advantage function $A(s,a)$ which is separate from $V(s)$ and $Q(s,a)$. You could write out a derivation for $A(s,a)$ that doesn't mention $V(s)$ or $Q(s,a)$ by name, but all the calculations necessary to estimate one or other or both will be inside it.

You may be able to design a solution that avoids explicitly calculating and maintaining $Q(s,a)$, but in that case you will not be able to avoid calculating $V(s)$.

$\endgroup$
0
$\begingroup$

In addition to Niel's answer, which is very much on point, I'd like to add that at least in principle, it is possible to compute an approximation to the advantage function $A(s,a)$ without having to compute both the value functions $V(s)$ and $Q(s,a)$ using separate networks, which I believe is the core of your question.

If you choose a baseline that is different from the value function, for instance, $b = \langle q(s,a) \rangle _a$ (mean q-value across actions), then you only need one network that computes the q-value and you will have the advantage function $A(s,a) = q(s,a) - b$ with the $b$ defined above. Of course, it is better to use the value function as the baseline, i.e. $b = E_{\pi(a|s)} q(s,a) = V(s)$. And though, in principle, you have access to $q(s,a)$ and $\pi(a|s)$ from which you can compute the $V(s)$, it is very tedious in practice because you need to separately fetch both the $q(s,a)$ and $\pi(a|s)$ from the respective networks, followed by manually combining them in order to compute the baseline $V(s)$.

Yet another way to compute an approximation to the advantage function could be in the classic spirit of policy-gradient methods. This would require neither the value network nor the q-value network. You could compute the rewards for many trajectories on policy conditioned on the taking action $a$ in state $s$ for all actions, and then compute the approximate advantage function with appropriate computations. As you might suspect, this is just not computationally efficient.

It just so happens that in practice, it is much better to have separate networks that learn $Q(s,a)$ and $V(s)$ in that it is less tedious and faster. The only downside is the redundancy of learning and representation, which is not much of an issue in practice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.