9
$\begingroup$

This question is in regards to the Deepmind paper on DDPG: https://arxiv.org/pdf/1509.02971v5.pdf.

Most (all?) implementations of the DDPG algorithm that I've seen compute the gradient update to the actor network by $\nabla(J)=\nabla_{\mu(s|\theta)}(Q(s,\mu(s|\theta))\nabla_{\theta}(\mu(s|\theta))$, where $\theta$ represents the actor network's parameters, $\mu$ represents the actor network, $Q$ repesents the critic network, and $s$ represents the state input. I'll call this equation 1.

Equation 1, as is shown in the paper, is derived by applying the chain rule to $\nabla(J)=\nabla_{\theta}(Q(s,\mu(s|\theta))$. This gives $ \nabla_{\mu(s|\theta)}(Q(s,\mu(s|\theta))\nabla_{\theta}(\mu(s|\theta))$.

My question is, using an auto-grad software package (Theano/Tensorflow/Torch/etc), is there any reason why I couldn't just compute the gradient of the output of $Q$ wrt $\theta$ directly? For some reason, all implementations seem to first compute the gradient of the output of $Q$ wrt $\mu(s)$ and then multiply it by the gradient of $\mu(s)$ wrt to $\theta$, per the chain rule. I don't understand why they do this--why not just directly compute the gradient of $Q$ wrt $\theta$ instead? Is there a reason you cannot do this? I.e, why do most updates seem to do this:

Q_grad = gradients( Q(s, mu(s|theta)), mu(s|theta) )
mu_grad = gradients( mu(s|theta), theta )
J_grad = Q_grad * mu_grad

Instead of this:

J_grad = gradients( Q(s, mu(s|theta)), theta )

Where the first input to "gradients" is the function you want to differentiate and the second input is what you are differentiating with respect to.

To be clear, I see no reason why $\nabla(J)=\nabla_{\theta}(Q)$ is a different update than equation 1, seeing as equation 1 is literally derived by applying the chain rule to $\nabla_{\theta}(Q)$, but I want to make sure I'm not missing some kind of subtlety.

$\endgroup$
  • $\begingroup$ Could you be more specific about what gradient you propose? I am unsure what you mean by "use the equation before applying chain rule as the update" $\endgroup$ – DaVinci Jan 30 '17 at 22:35
  • $\begingroup$ @DaVinci sorry for the ambiguity! I updated the original post to (hopefully) be clearer. $\endgroup$ – Bill Feb 1 '17 at 3:54
6
$\begingroup$

There is no difference in the calculation. I was wondering the same thing and verified in my own TensorFlow DDPG implementation by trying both and asserting that the numerical values are identical. As expected, they are.

I noticed that most tutorial-like implementations (e.g. Patrick Emami's) explicitly show the multiplication. However, OpenAI's baselines implementation $does$ directly compute $\nabla_{\theta^\mu} Q$. (They do this by defining a loss on the actor network equal to $-\nabla_{\theta^\mu} Q$, averaged across the batch).

There is one reason that you'd want to separate out $\nabla_a Q$ from $\nabla_{\theta^\mu} \mu$ and multiply them. This is if you want to directly manipulate one of the terms. For example, Hausknecht and Stone do "inverting gradients" on $\nabla_a Q$ to coerce actions to stay within the environment's range.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

This way you can define two independent networks. Otherwise, you may have to define a large network and distinguish which part belongs to the policy and which one to the state-action value function.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I am not sure to understand this multiplication between the two gradient terms..

When you compute this using say tensorflow:

J_grad = gradients( Q(s, mu(s|theta)), theta )

It applies the chain-rule and therefore computes the gadients of Q w.r.t the output of the policy network mu(s|theta) and then backpropoagates those "errors" through the policy network to obtain the sampled gradients w.r.t theta (the parameters of every layer of your policy network).

However, when you do:

Q_grad = gradients( Q(s, mu(s|theta)), mu(s|theta) )
mu_grad = gradients( mu(s|theta), theta )
J_grad = Q_grad * mu_grad

Then in my understanding, it (1) computes the gradient of Q w.r.t the ouput of the policy network mu(s|theta) and (2) the gradient of the output of the policy network mu(s|theta) w.r.t the policy parameters theta, but this time SEPARATELY. What I don't understand is that now, you have on one hand your first gradient which is a vector of size (1, action_dim) and on the other, you have your second gradient which is a vector of size (1, theta_dim). To apply your update, you need a gradient w.r.t to theta which would be a vector of size (1, theta_dim). So what exactly is this multiplication doing in the third line, and how is it equivalent to backpropagating the first gradient through the policy network:

J_grad = Q_grad * mu_grad

Question:

Does it just perform an outer-product creating a matrix of shape (action_dim, theta_dim) and then reduced by summing over the dimension to obtain our update vector of shape (1, theta) ? If so, why is this valid (equivalent to backpropagating the first gradient through the policy network)?

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.