This question is in regards to the Deepmind paper on DDPG: https://arxiv.org/pdf/1509.02971v5.pdf.

Most (all?) implementations of the DDPG algorithm that I've seen compute the gradient update to the actor network by $\nabla(J)=\nabla_{\mu(s|\theta)}(Q(s,\mu(s|\theta))\nabla_{\theta}(\mu(s|\theta))$, where $\theta$ represents the actor network's parameters, $\mu$ represents the actor network, $Q$ repesents the critic network, and $s$ represents the state input. I'll call this equation 1.

Equation 1, as is shown in the paper, is derived by applying the chain rule to $\nabla(J)=\nabla_{\theta}(Q(s,\mu(s|\theta)) = \nabla_{\mu(s|\theta)}(Q(s,\mu(s|\theta))\nabla_{\theta}(\mu(s|\theta))$.

My question is, using an auto-grad software package (Theano/Tensorflow/Torch/etc), is there any reason why I couldn't just compute the gradient of the output of $Q$ wrt $\theta$ directly? For some reason, all implementations seem to first compute the gradient of the output of $Q$ wrt $\mu(s)$ and then multiply it by the graident of $\mu(s)$ wrt to $\theta$, per the chain rule. I don't understand why they do this--why not just directly compute the gradient of $Q$ wrt $\theta$ instead? Is there a reason you cannot do this? I.e, why do most updates seem to do this:

Q_grad = gradients( Q(s, mu(s|theta)), mu(s|theta) )
mu_grad = gradients( mu(s|theta), theta )
J_grad = Q_grad * mu_grad

Instead of this:

J_grad = gradients( Q(s, mu(s|theta)), theta )

Where the first input to "gradients" is the function you want to differentiate and the second input is what you are differentiating with respect to.

To be clear, I see no reason why $\nabla(J)=\nabla_{\theta}(Q)$ is a different update than equation 1, seeing as equation 1 is literally derived by applying the chain rule to $\nabla_{\theta}(Q)$, but I want to make sure I'm not missing some kind of subtlety.