Computing the Actor Gradient Update in the Deep Deterministic Policy Gradient (DDPG) algorithm

Question

This question is in regards to the Deepmind paper on DDPG: https://arxiv.org/pdf/1509.02971v5.pdf.

Most (all?) implementations of the DDPG algorithm that I've seen compute the gradient update to the actor network by $\nabla(J)=\nabla_{\mu(s|\theta)}(Q(s,\mu(s|\theta))\nabla_{\theta}(\mu(s|\theta))$, where $\theta$ represents the actor network's parameters, $\mu$ represents the actor network, $Q$ repesents the critic network, and $s$ represents the state input. I'll call this equation 1.

Equation 1, as is shown in the paper, is derived by applying the chain rule to $\nabla(J)=\nabla_{\theta}(Q(s,\mu(s|\theta))$. This gives $ \nabla_{\mu(s|\theta)}(Q(s,\mu(s|\theta))\nabla_{\theta}(\mu(s|\theta))$.

My question is, using an auto-grad software package (Theano/Tensorflow/Torch/etc), is there any reason why I couldn't just compute the gradient of the output of $Q$ wrt $\theta$ directly? For some reason, all implementations seem to first compute the gradient of the output of $Q$ wrt $\mu(s)$ and then multiply it by the gradient of $\mu(s)$ wrt to $\theta$, per the chain rule. I don't understand why they do this--why not just directly compute the gradient of $Q$ wrt $\theta$ instead? Is there a reason you cannot do this? I.e, why do most updates seem to do this:

Q_grad = gradients( Q(s, mu(s|theta)), mu(s|theta) )
mu_grad = gradients( mu(s|theta), theta )
J_grad = Q_grad * mu_grad

Instead of this:

J_grad = gradients( Q(s, mu(s|theta)), theta )

Where the first input to "gradients" is the function you want to differentiate and the second input is what you are differentiating with respect to.

To be clear, I see no reason why $\nabla(J)=\nabla_{\theta}(Q)$ is a different update than equation 1, seeing as equation 1 is literally derived by applying the chain rule to $\nabla_{\theta}(Q)$, but I want to make sure I'm not missing some kind of subtlety.

Answer 1

There is no difference in the calculation. I was wondering the same thing and verified in my own TensorFlow DDPG implementation by trying both and asserting that the numerical values are identical. As expected, they are.

I noticed that most tutorial-like implementations (e.g. Patrick Emami's) explicitly show the multiplication. However, OpenAI's baselines implementation $does$ directly compute $\nabla_{\theta^\mu} Q$. (They do this by defining a loss on the actor network equal to $-\nabla_{\theta^\mu} Q$, averaged across the batch).

There is one reason that you'd want to separate out $\nabla_a Q$ from $\nabla_{\theta^\mu} \mu$ and multiply them. This is if you want to directly manipulate one of the terms. For example, Hausknecht and Stone do "inverting gradients" on $\nabla_a Q$ to coerce actions to stay within the environment's range.

Answer 2

This way you can define two independent networks. Otherwise, you may have to define a large network and distinguish which part belongs to the policy and which one to the state-action value function.

Answer 3

I am not sure to understand this multiplication between the two gradient terms..

When you compute this using say tensorflow:

J_grad = gradients( Q(s, mu(s|theta)), theta )

It applies the chain-rule and therefore computes the gadients of Q w.r.t the output of the policy network mu(s|theta) and then backpropoagates those "errors" through the policy network to obtain the sampled gradients w.r.t theta (the parameters of every layer of your policy network).

However, when you do:

Q_grad = gradients( Q(s, mu(s|theta)), mu(s|theta) )
mu_grad = gradients( mu(s|theta), theta )
J_grad = Q_grad * mu_grad

Then in my understanding, it (1) computes the gradient of Q w.r.t the ouput of the policy network mu(s|theta) and (2) the gradient of the output of the policy network mu(s|theta) w.r.t the policy parameters theta, but this time SEPARATELY. What I don't understand is that now, you have on one hand your first gradient which is a vector of size (1, action_dim) and on the other, you have your second gradient which is a vector of size (1, theta_dim). To apply your update, you need a gradient w.r.t to theta which would be a vector of size (1, theta_dim). So what exactly is this multiplication doing in the third line, and how is it equivalent to backpropagating the first gradient through the policy network:

J_grad = Q_grad * mu_grad

Question:

Does it just perform an outer-product creating a matrix of shape (action_dim, theta_dim) and then reduced by summing over the dimension to obtain our update vector of shape (1, theta) ? If so, why is this valid (equivalent to backpropagating the first gradient through the policy network)?