I am trying to find the Hessian matrix for the maximum log likelihood function given training data {(xi, yi)} for i=1:N with yi ∈ {+1, −1} for each i = 1, . . . , N for the function: enter image description here

When I try to find the Hessian matrix for this function, I get this: enter image description here

which gives a vector, not a number which doesn't seem be correct. How do you calculate the off diagonals of the Hessian matrix without the result being a vector?

This is what I get for the diagonals of the Hessian and the gradients:

enter image description here

  • 1
    $\begingroup$ as far as I can see you have only differentiated once with respect to the parameter (w,b) hence you get a vector. Then you need to differentiate again with respect to (w,b). Also there are other errors as I can see from a quick glance (might be wrong) but have another look. $\endgroup$ – Jesper Hybel Dec 5 '18 at 22:58
  • 1
    $\begingroup$ The factors of $\log(10)$ definitely don't belong: "log" in the log likelihood formula means the natural log. $\endgroup$ – whuber Dec 5 '18 at 23:10

...which gives a vector, not a number...

If you differentiate a multivariable function $f$ with respect to a vector $\mathbf{w}$ then you get a derivative that is itself a vector (the gradient vector). If you would like to get scalar second derivatives then you need to differentiate with respect to the elements of $\mathbf{w}$ instead of the whole vector. In this particular case, if you have a vector $\mathbf{w} = (w_1,...,w_m)$ then the Hessian matrix will consist of the following scalar second-order partial derivatives:

$$\mathbf{H}(b,\mathbf{w}) = \begin{bmatrix} \frac{\partial^2 f}{\partial b^2}(b, \mathbf{w}) & \frac{\partial^2 f}{\partial b \partial w_1}(b, \mathbf{w}) & \cdots & \frac{\partial^2 f}{\partial b \partial w_m}(b, \mathbf{w}) \\ \frac{\partial^2 f}{\partial b \partial w_1}(b, \mathbf{w}) & \frac{\partial^2 f}{\partial w_1^2}(b, \mathbf{w}) & \cdots & \frac{\partial^2 f}{\partial w_1 \partial w_m}(b, \mathbf{w}) \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial b \partial w_m}(b, \mathbf{w}) & \frac{\partial^2 f}{\partial w_1 \partial w_m}(b, \mathbf{w}) & \cdots & \frac{\partial^2 f}{\partial w_m^2}(b, \mathbf{w}) \\ \end{bmatrix}.$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.