On p251 of Bishop's machine learning book, the Hessian for least squares is derived (as a preliminary step to the outer product approximation):

$ E = \frac{1}{2} \sum_{n=1}^{N} \left(y_n - t_n\right)^2$

$H = \nabla \nabla E = \sum_{n=1}^{N} \nabla y_n \left(\nabla y_n\right)^T + \sum_{n=1}^{N} (y_n - t_n) \nabla \nabla y_n $

  • Firstly, why is the Hessian not given by $\nabla \nabla ^T E$?
  • Secondly, could someone please explain how the full expression for the Hessian is obtained?
  • $\begingroup$ In the book I don't see an explicit transpose as in $(\nabla y_n)^T$. $\endgroup$
    – Gilles
    Feb 1 '16 at 12:24
  • $\begingroup$ @Gilles you must be reading a pre-2009 edition (missing transpose is in errata list). The above is as written in the corrected 3rd printing. $\endgroup$
    – Tarrare
    Feb 1 '16 at 13:01

Well, I think the use of the operator is applying $\nabla$ operator twice, one time and one time again on the function $E$. But ... It is often written confusingly as $\nabla^2$, and in this case is $\nabla\left(\nabla E\right)$. Or since $\nabla$ is a vector, $\nabla\nabla^T$ can work as a matrix operator on $E$, but you're just applying the $\nabla$ operator twice.

For the expression, you have: \begin{align} H &= \nabla \left(\nabla E\right) = \nabla\left[\nabla \left(\frac 12 \sum_{n=1}^{N} (y_n - t_n)^2\right)\right]\\ &= \nabla \left(\sum_{n=1}^{N} (y_n - t_n) \nabla y_n \right)\\ &=\sum_{n=1}^{N} \nabla y_n \left(\nabla y_n \right) + \sum_{n=1}^{N} (y_n - t_n)\nabla \left(\nabla y_n\right) \end{align}

I still don't get why Bishop has the transpose in his latter printings.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.