3
$\begingroup$

I started coding a Dueling Network Architectures for Deep Reinforcement Learning. I devided my network into two streams, arriving at a V(s) value and A(s,a) values. I arrived at the Q(s,a) output values by calculating: Q(s,a) = V(s) + ( A(s,a) - Mean of All A(s,a) )

Now my problem is with backpropagation. After calculating the loss, how do I calculate the gradients of V(s) and A(s,a) without there being any weights?

$\endgroup$
6
  • $\begingroup$ I don't understand the question. Both V and A are networks and therefore have weights. $\endgroup$
    – DaVinci
    Nov 30, 2016 at 16:41
  • $\begingroup$ but when we get to V and A, how do we get to Q? through weights? or through the calculation above? $\endgroup$
    – naimelhajj
    Nov 30, 2016 at 17:22
  • $\begingroup$ through the calculation above. There's no part of backpropagation that requires that all layers have weights though. Pooling layers don't, for example. $\endgroup$
    – DaVinci
    Nov 30, 2016 at 19:38
  • $\begingroup$ @DaVinci Ah sorry I wasn't aware of that. So how do you train the network then? What I know to do is use the gradient to adjust weights. (beginner, obviously) $\endgroup$
    – naimelhajj
    Nov 30, 2016 at 20:26
  • $\begingroup$ Just implement backprop and you get the gradient for the weights of both V and A. Or, if you implement it in a library such as theano or tensorflow, simply do the Q(s, a) = V(s) + (A(s,a) - mean_a A(s,a)) in theano or tensorflow operations and it will get the gradients for you. You'd have a V tensor and an A tensor and you can define a Q tensor via, Q = V + A + T.mean(V, axis=1). $\endgroup$
    – DaVinci
    Nov 30, 2016 at 21:05

2 Answers 2

3
$\begingroup$

The gradient in DQN is given by

enter image description here

Since Q is a simple sum of functions you have:

$$\nabla_{\theta} Q(s,a) = \nabla_{\theta}V(s) + \nabla{\theta}A(s,a) - \frac{1}{numActions} \sum_{a'}\nabla_{\theta}A(s, a')$$

You get the gradients of the V and A networks as usual by backprop.

$\endgroup$
2
  • $\begingroup$ Could we say that Delta(v) = V(s) * ( 1 - V(s) ) * Sum of Deltas Q(s,a) ? If so, what about Deltas of A(s,a)? $\endgroup$
    – naimelhajj
    Dec 4, 2016 at 19:50
  • $\begingroup$ I'm not sure what you mean. But you don't need to calculate the deltas separately since you can simply calculate the Q values given V and A. $\endgroup$
    – DaVinci
    Dec 5, 2016 at 18:39
0
$\begingroup$

Now my problem is with backpropagation. After calculating the loss, how do I calculate the gradients of V(s) and A(s,a) without there being any weights?

If so, what about Deltas of A(s,a)

Looking at the comments from the currently accepted answer, I'd like to add mine too.

This post from Data Science shows the gradient vector that should enter V and A layers respectively.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.