# Backpropagating a Dueling Architecture Network: Gradient Calculation

I started coding a Dueling Network Architectures for Deep Reinforcement Learning. I devided my network into two streams, arriving at a V(s) value and A(s,a) values. I arrived at the Q(s,a) output values by calculating: Q(s,a) = V(s) + ( A(s,a) - Mean of All A(s,a) )

Now my problem is with backpropagation. After calculating the loss, how do I calculate the gradients of V(s) and A(s,a) without there being any weights?

• I don't understand the question. Both V and A are networks and therefore have weights. Nov 30, 2016 at 16:41
• but when we get to V and A, how do we get to Q? through weights? or through the calculation above? Nov 30, 2016 at 17:22
• through the calculation above. There's no part of backpropagation that requires that all layers have weights though. Pooling layers don't, for example. Nov 30, 2016 at 19:38
• @DaVinci Ah sorry I wasn't aware of that. So how do you train the network then? What I know to do is use the gradient to adjust weights. (beginner, obviously) Nov 30, 2016 at 20:26
• Just implement backprop and you get the gradient for the weights of both V and A. Or, if you implement it in a library such as theano or tensorflow, simply do the Q(s, a) = V(s) + (A(s,a) - mean_a A(s,a)) in theano or tensorflow operations and it will get the gradients for you. You'd have a V tensor and an A tensor and you can define a Q tensor via, Q = V + A + T.mean(V, axis=1). Nov 30, 2016 at 21:05

The gradient in DQN is given by

Since Q is a simple sum of functions you have:

$$\nabla_{\theta} Q(s,a) = \nabla_{\theta}V(s) + \nabla{\theta}A(s,a) - \frac{1}{numActions} \sum_{a'}\nabla_{\theta}A(s, a')$$

You get the gradients of the V and A networks as usual by backprop.

• Could we say that Delta(v) = V(s) * ( 1 - V(s) ) * Sum of Deltas Q(s,a) ? If so, what about Deltas of A(s,a)? Dec 4, 2016 at 19:50
• I'm not sure what you mean. But you don't need to calculate the deltas separately since you can simply calculate the Q values given V and A. Dec 5, 2016 at 18:39

Now my problem is with backpropagation. After calculating the loss, how do I calculate the gradients of V(s) and A(s,a) without there being any weights?

If so, what about Deltas of A(s,a)