# Outer product approximation of the Hessian

On p251 of Bishop's machine learning book, the Hessian for least squares is derived (as a preliminary step to the outer product approximation):

$E = \frac{1}{2} \sum_{n=1}^{N} \left(y_n - t_n\right)^2$

$H = \nabla \nabla E = \sum_{n=1}^{N} \nabla y_n \left(\nabla y_n\right)^T + \sum_{n=1}^{N} (y_n - t_n) \nabla \nabla y_n$

• Firstly, why is the Hessian not given by $\nabla \nabla ^T E$?
• Secondly, could someone please explain how the full expression for the Hessian is obtained?
• In the book I don't see an explicit transpose as in $(\nabla y_n)^T$. – Gilles Feb 1 '16 at 12:24
• @Gilles you must be reading a pre-2009 edition (missing transpose is in errata list). The above is as written in the corrected 3rd printing. – Tarrare Feb 1 '16 at 13:01

Well, I think the use of the operator is applying $\nabla$ operator twice, one time and one time again on the function $E$. But ... It is often written confusingly as $\nabla^2$, and in this case is $\nabla\left(\nabla E\right)$. Or since $\nabla$ is a vector, $\nabla\nabla^T$ can work as a matrix operator on $E$, but you're just applying the $\nabla$ operator twice.