1
$\begingroup$

The BFGS is an algorithm trying to approximate the Hessian matrix in NewtonRaphson Method. I read about the BFGSMultiply Method trying to approximate the Hessian without dealing with the Hessian explicitly, while resulting an ever growing memory demand for storing the history data of the difference, which spawns the LBFGS.

My question is, why not dealing with the Hessian in the BFGS directly to save the memory?

$\endgroup$
2
$\begingroup$

The BFGS Hessian approximation can either be based on the full history of gradients, in which case it is referred to as BFGS, or it can be based only on the most recent m gradients, in which case it is known as limited memory BFGS, abbreviated as L-BFGS.

The advantage of L-BFGS is that is requires only retaining the most recent m gradients, where m is usually around 10 to 20, which is a much smaller storage requirement than n*(n+1)/2 elements required to store the full (triangle) of a Hessian estimate, as in required with BFGS, where n is the problem dimension. Unlike (full) BFGS, the estimate of the Hessian is never explicitly formed or stored in L-BFGS; rather, the calculations which would be required with the estimate of the Hessian are accomplished without explicitly forming it.

L-BFGS is used instead of BFGS for very large problems (when n is very large), but might not perform as well as BFGS. Therefore, BFGS is preferred over L-BFGS when the memory requirements of BFGS can be met. On the other hand, L-BFGS may not be much worse in performance than BFGS.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.