Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
On p251 of Bishop's machine learning book, the Hessian for least squares is derived (as a preliminary step to the outer product approximation):
$ E = \frac{1}{2} \sum_{n=1}^{N} \left(y_n - t_n\right)^2$
$H = \nabla \nabla E = \sum_{n=1}^{N} \nabla y_n \left(\nabla y_n\right)^T + \sum_{n=1}^{N} (y_n - t_n) \nabla \nabla y_n $
Well, I think the use of the operator is applying $\nabla$ operator twice, one time and one time again on the function $E$. But ... It is often written confusingly as $\nabla^2$, and in this case is $\nabla\left(\nabla E\right)$. Or since $\nabla$ is a vector, $\nabla\nabla^T$ can work as a matrix operator on $E$, but you're just applying the $\nabla$ operator twice.
For the expression, you have: \begin{align} H &= \nabla \left(\nabla E\right) = \nabla\left[\nabla \left(\frac 12 \sum_{n=1}^{N} (y_n - t_n)^2\right)\right]\\ &= \nabla \left(\sum_{n=1}^{N} (y_n - t_n) \nabla y_n \right)\\ &=\sum_{n=1}^{N} \nabla y_n \left(\nabla y_n \right) + \sum_{n=1}^{N} (y_n - t_n)\nabla \left(\nabla y_n\right) \end{align}
I still don't get why Bishop has the transpose in his latter printings.
