# Derivative of $\nabla_{\theta} f(x, \theta) f(x, \theta)$ (the gradient of the function times the function itself)

I am having troubles computing the derivative of $$\nabla_{\theta}f(x, \theta)f(x, \theta)$$ (the gradient of the function $$f(x, \theta)$$ times the function itself) that is \begin{align} D(\nabla_{\theta}f(x, \theta)f(x, \theta)), \end{align} for $$\theta = (\theta_1, \theta_2)^T$$ and where $$Df = (\partial_{\theta_1}f(x, \theta), \partial_{\theta_2}f(x, \theta)) \in \mathbb{R}^{1\times2}$$. The gradient is a column vector, $$\nabla_{\theta}f(x, \theta) \in \mathbb{R}^{2\times1}$$.

I get this to be equal to \begin{align} D^2f(x, \theta)f(x, \theta) + \nabla_{\theta}f(x, \theta)\nabla_{\theta}f(x, \theta)^T, \end{align} where $$D^2f(x, \theta)$$ is the Hessian and $$\nabla_{\theta}f(x, \theta)\nabla_{\theta}f(x, \theta)^T$$ is the outer product of the gradients, but my numerical calculations (using the numDeriv package in R) do not seem to match this.

Where am I going wrong?

• When you say $D^2f(x,\theta)f(x,\theta)$ is the Hessian, is that just a typo for $D^2f(x,\theta)$ is the Hessian? Oct 20, 2021 at 21:11
• @ThomasLumley yes, sorry for that. Oct 20, 2021 at 21:34
• First, remove the superfluous references to "$x$" because they complicate the problem. Then, consider a simple function that is easy to differentiate several times, such as $f(\theta)=\theta_1^2+2\theta_1\theta_2 + 3\theta_2^2.$ It ought to be quick and easy to work out the answer--and that will point the way towards a general answer.
– whuber
Oct 20, 2021 at 22:29
• thanks @whuber! Oct 21, 2021 at 10:31

I figured it out myself. Thanks for the function recommendation, @whuber.

For the function $$f(\theta) = \theta_1^2 + \theta_2^2 + \theta_1 \theta_2$$, the gradient of the function equals \begin{align} \nabla_{\theta}f(\theta) = \left[\begin{matrix}2 \theta_{1} + \theta_{2}\\\theta_{1} + 2 \theta_{2}\end{matrix}\right], \end{align} and the Hessian equals \begin{align} D^2f(\theta) = \left[\begin{matrix}2 & 1\\1 & 2\end{matrix}\right]. \end{align}

Now, the gradient of the function times itself equals

\begin{align} \nabla_{\theta}f(\theta) f(\theta) = \left[\begin{matrix}\left(2 \theta_{1} + \theta_{2}\right) \left(\theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2}\right)\\\left(\theta_{1} + 2 \theta_{2}\right) \left(\theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2}\right)\end{matrix}\right]. \end{align}

The derivative of this quantity (the one I was interested in) equals

\begin{align} D(\nabla_{\theta}f(\theta) f(\theta)) &= \left[\begin{matrix}2 \theta_{1}^{2} + 2 \theta_{1} \theta_{2} + 2 \theta_{2}^{2} + \left(2 \theta_{1} + \theta_{2}\right)^{2} & \theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2} + \left(\theta_{1} + 2 \theta_{2}\right) \left(2 \theta_{1} + \theta_{2}\right)\\\theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2} + \left(\theta_{1} + 2 \theta_{2}\right) \left(2 \theta_{1} + \theta_{2}\right) & 2 \theta_{1}^{2} + 2 \theta_{1} \theta_{2} + 2 \theta_{2}^{2} + \left(\theta_{1} + 2 \theta_{2}\right)^{2}\end{matrix}\right] \\ &= \left[\begin{matrix}2 & 1\\1 & 2\end{matrix}\right](\theta_1^2 + \theta_2^2 + \theta_1 \theta_2) + \left[\begin{matrix}2 \theta_{1} + \theta_{2}\\\theta_{1} + 2 \theta_{2}\end{matrix}\right] \left[\begin{matrix}2 \theta_{1} + \theta_{2}, \theta_{1} + 2 \theta_{2}\end{matrix}\right] \\ & = D^2f(\theta)f(\theta) + \nabla_{\theta}f(\theta) \nabla_{\theta}f(\theta)^T. \end{align}

So the identity holds.

For a general function $$f{\left(\theta_{1},\theta_{2} \right)}$$, the gradient of this function times itself equals \begin{align} \left[\begin{matrix}f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\\f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] \end{align}

and the derivative of this equals \begin{align} \left[\begin{matrix}f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{1}^{2}} f{\left(\theta_{1},\theta_{2} \right)} + \left(\frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\right)^{2} & f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} + \frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\\f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} + \frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)} & f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{2}^{2}} f{\left(\theta_{1},\theta_{2} \right)} + \left(\frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\right)^{2}\end{matrix}\right] \end{align}

which equals \begin{align} \left[\begin{matrix}\frac{\partial^{2}}{\partial \theta_{1}^{2}} f{\left(\theta_{1},\theta_{2} \right)} & \frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\\\frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} & \frac{\partial^{2}}{\partial \theta_{2}^{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] f{\left(\theta_{1},\theta_{2} \right)} + \left[\begin{matrix}\frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\\\frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] \left[\begin{matrix}\frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}, \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] \\ = D^2f{\left(\theta_{1},\theta_{2} \right)} + \nabla_{\theta}f{\left(\theta_{1},\theta_{2} \right)}f{\left(\theta_{1},\theta_{2} \right)}^T. \end{align}

The reason for my confusion (I already did those derivations in my notebook) was a minor typo in my R code :) I leave the derivations here for someone in the future who might run into the same problem as me.

The reason I was interested in $$D(\nabla_{\theta}f{\left(\theta_{1},\theta_{2} \right)}f{\left(\theta_{1},\theta_{2} \right)})$$ is when finding the Hessian of the loss function in non-linear least squares \begin{align} \frac{1}{2}(y - f{\left(\theta_{1},\theta_{2} \right)})^2. \end{align}