On Jorge Nocedal's , Page 501, "This property alone is enough to make many unconstrained minimization algorithms such as quasi-Newton and conjugate gradient perform poorly. Newton’s method, on the other hand, is not sensitive to ill conditioning of the Hessian". Can any one give a more detailed analysis?

  • $\begingroup$ Hi: Newton's method doesn't use any information regarding the second derivative so that's why it's insensitive to it. It's a method that only relies on the first derivative. This is not the case for quasi-Newton and conjugate gradient methods. $\endgroup$
    – mlofton
    Jan 8 '19 at 6:23
  • 6
    $\begingroup$ @mlofton: ?????????????????????? $\endgroup$ Jan 8 '19 at 9:26
  • $\begingroup$ @kjeitil b halvorsen: I thought what I said was the case. if not, I'll delete comment. $\endgroup$
    – mlofton
    Jan 9 '19 at 15:37
  • 3
    $\begingroup$ @mlofton Newton's method is a second order optimization method, which means it does in fact use the second derivative and as such computes (and inverts) the hessian directly. Newton-CG on the other hand solves a system of equations to 'invert' the hessian. All second order methods use the Hessian or at least some approximation. $\endgroup$
    – Avelina
    Jun 28 at 15:59
  • $\begingroup$ oops: my mistake. I thought it just used the first derivative. thanks for heads up. should I delete my totally wrong comment-statement. $\endgroup$
    – mlofton
    Jun 28 at 18:11

"An ill conditioned hessian means that the gradient is changing direction rapidly; this causes a problem because our method is only looking at the gradient at one point.􏰂 Newton's method is solving this problem because it looks at the hessian and is taking the rate of change of the gradient into account."

source: https://ocw.mit.edu/courses/sloan-school-of-management/15-084j-nonlinear-programming-spring-2004/recitations/rec3.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.