1
$\begingroup$

I am having troubles computing the derivative of $\nabla_{\theta}f(x, \theta)f(x, \theta) $ (the gradient of the function $f(x, \theta)$ times the function itself) that is \begin{align} D(\nabla_{\theta}f(x, \theta)f(x, \theta)), \end{align} for $\theta = (\theta_1, \theta_2)^T$ and where $Df = (\partial_{\theta_1}f(x, \theta), \partial_{\theta_2}f(x, \theta)) \in \mathbb{R}^{1\times2}$. The gradient is a column vector, $\nabla_{\theta}f(x, \theta) \in \mathbb{R}^{2\times1}$.

I get this to be equal to \begin{align} D^2f(x, \theta)f(x, \theta) + \nabla_{\theta}f(x, \theta)\nabla_{\theta}f(x, \theta)^T, \end{align} where $D^2f(x, \theta)$ is the Hessian and $\nabla_{\theta}f(x, \theta)\nabla_{\theta}f(x, \theta)^T$ is the outer product of the gradients, but my numerical calculations (using the numDeriv package in R) do not seem to match this.

Where am I going wrong?

$\endgroup$
4
  • $\begingroup$ When you say $D^2f(x,\theta)f(x,\theta)$ is the Hessian, is that just a typo for $D^2f(x,\theta)$ is the Hessian? $\endgroup$ Oct 20, 2021 at 21:11
  • $\begingroup$ @ThomasLumley yes, sorry for that. $\endgroup$ Oct 20, 2021 at 21:34
  • $\begingroup$ First, remove the superfluous references to "$x$" because they complicate the problem. Then, consider a simple function that is easy to differentiate several times, such as $f(\theta)=\theta_1^2+2\theta_1\theta_2 + 3\theta_2^2.$ It ought to be quick and easy to work out the answer--and that will point the way towards a general answer. $\endgroup$
    – whuber
    Oct 20, 2021 at 22:29
  • 1
    $\begingroup$ thanks @whuber! $\endgroup$ Oct 21, 2021 at 10:31

1 Answer 1

3
$\begingroup$

I figured it out myself. Thanks for the function recommendation, @whuber.

For the function $f(\theta) = \theta_1^2 + \theta_2^2 + \theta_1 \theta_2$, the gradient of the function equals \begin{align} \nabla_{\theta}f(\theta) = \left[\begin{matrix}2 \theta_{1} + \theta_{2}\\\theta_{1} + 2 \theta_{2}\end{matrix}\right], \end{align} and the Hessian equals \begin{align} D^2f(\theta) = \left[\begin{matrix}2 & 1\\1 & 2\end{matrix}\right]. \end{align}

Now, the gradient of the function times itself equals

\begin{align} \nabla_{\theta}f(\theta) f(\theta) = \left[\begin{matrix}\left(2 \theta_{1} + \theta_{2}\right) \left(\theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2}\right)\\\left(\theta_{1} + 2 \theta_{2}\right) \left(\theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2}\right)\end{matrix}\right]. \end{align}

The derivative of this quantity (the one I was interested in) equals

\begin{align} D(\nabla_{\theta}f(\theta) f(\theta)) &= \left[\begin{matrix}2 \theta_{1}^{2} + 2 \theta_{1} \theta_{2} + 2 \theta_{2}^{2} + \left(2 \theta_{1} + \theta_{2}\right)^{2} & \theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2} + \left(\theta_{1} + 2 \theta_{2}\right) \left(2 \theta_{1} + \theta_{2}\right)\\\theta_{1}^{2} + \theta_{1} \theta_{2} + \theta_{2}^{2} + \left(\theta_{1} + 2 \theta_{2}\right) \left(2 \theta_{1} + \theta_{2}\right) & 2 \theta_{1}^{2} + 2 \theta_{1} \theta_{2} + 2 \theta_{2}^{2} + \left(\theta_{1} + 2 \theta_{2}\right)^{2}\end{matrix}\right] \\ &= \left[\begin{matrix}2 & 1\\1 & 2\end{matrix}\right](\theta_1^2 + \theta_2^2 + \theta_1 \theta_2) + \left[\begin{matrix}2 \theta_{1} + \theta_{2}\\\theta_{1} + 2 \theta_{2}\end{matrix}\right] \left[\begin{matrix}2 \theta_{1} + \theta_{2}, \theta_{1} + 2 \theta_{2}\end{matrix}\right] \\ & = D^2f(\theta)f(\theta) + \nabla_{\theta}f(\theta) \nabla_{\theta}f(\theta)^T. \end{align}

So the identity holds.

For a general function $f{\left(\theta_{1},\theta_{2} \right)}$, the gradient of this function times itself equals \begin{align} \left[\begin{matrix}f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\\f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] \end{align}

and the derivative of this equals \begin{align} \left[\begin{matrix}f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{1}^{2}} f{\left(\theta_{1},\theta_{2} \right)} + \left(\frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\right)^{2} & f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} + \frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\\f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} + \frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)} & f{\left(\theta_{1},\theta_{2} \right)} \frac{\partial^{2}}{\partial \theta_{2}^{2}} f{\left(\theta_{1},\theta_{2} \right)} + \left(\frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\right)^{2}\end{matrix}\right] \end{align}

which equals \begin{align} \left[\begin{matrix}\frac{\partial^{2}}{\partial \theta_{1}^{2}} f{\left(\theta_{1},\theta_{2} \right)} & \frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\\\frac{\partial^{2}}{\partial \theta_{2}\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)} & \frac{\partial^{2}}{\partial \theta_{2}^{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] f{\left(\theta_{1},\theta_{2} \right)} + \left[\begin{matrix}\frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}\\\frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] \left[\begin{matrix}\frac{\partial}{\partial \theta_{1}} f{\left(\theta_{1},\theta_{2} \right)}, \frac{\partial}{\partial \theta_{2}} f{\left(\theta_{1},\theta_{2} \right)}\end{matrix}\right] \\ = D^2f{\left(\theta_{1},\theta_{2} \right)} + \nabla_{\theta}f{\left(\theta_{1},\theta_{2} \right)}f{\left(\theta_{1},\theta_{2} \right)}^T. \end{align}

The reason for my confusion (I already did those derivations in my notebook) was a minor typo in my R code :) I leave the derivations here for someone in the future who might run into the same problem as me.

The reason I was interested in $D(\nabla_{\theta}f{\left(\theta_{1},\theta_{2} \right)}f{\left(\theta_{1},\theta_{2} \right)})$ is when finding the Hessian of the loss function in non-linear least squares \begin{align} \frac{1}{2}(y - f{\left(\theta_{1},\theta_{2} \right)})^2. \end{align}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.