Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
The purpose of the paper was to optimize some parameters by maximizing the regularized log-likelihood. Then they calculate Partial derivatives. And then authors mention that they optimize the equation using L-BFGS, a standard quasi-Newton procedure to optimize smooth functions of many variables (no more details).
How does it work ?
Basically think of L-BFGS as a way of finding a (local) minimum of an objective function, making use of objective function values and the gradient of the objective function. That level of description covers many optimization methods in addition to L-BFGS though. You can read more about it in section 7.2 of Nocedal and Wright "Numerical Optimization, 2nd edition" http://www.springer.com/us/book/9780387303031 . A very cursory discussion of L-BFGS is provided at https://en.wikipedia.org/wiki/Limited-memory_BFGS .
First order method means gradients (first derivatives) (and maybe objective function values) are used, but not Hessian (second derivatives). Think of, for instance, gradient descent and steepest descent, among many others.
Second order method means gradients and Hessian are used (and maybe objective function values). Second order methods can be either based on
"Exact" Hessian matrix (or finite differences of gradients), in which case they are known as Newton methods or
Quasi-Newton methods, which approximate the Hessian based on differences of gradients over several iterations, by imposing a "secant" (Quasi-Newton) condition. There are many different Quasi-Newton methods, which estimate the Hessian in different ways. One of the most popular is BFGS. The BFGS Hessian approximation can either be based on the full history of gradients, in which case it is referred to as BFGS, or it can be based only on the most recent m gradients, in which case it is known as limited memory BFGS, abbreviated as L-BFGS. The advantage of L-BFGS is that is requires only retaining the most recent m gradients, where m is usually around 5 to 20, which is a much smaller storage requirement than n*(n+1)/2 elements required to store the full (triangle) of a Hessian estimate, as is required with BFGS, where n is the problem dimension. Unlike (full) BFGS, the estimate of the Hessian is never explicitly formed or stored in L-BFGS (although some implementations of BFGS only form and update the Choelsky factor of the Hessian approximation, rather than the Hessian approximation itself); rather, the calculations which would be required with the estimate of the Hessian are accomplished without explicitly forming it. L-BFGS is used instead of BFGS for very large problems (when n is very large), but might not perform as well as BFGS. Therefore, BFGS is preferred over L-BFGS when the memory requirements of BFGS can be met. On the other hand, L-BFGS may not be much worse in performance than BFGS.
Even at this level of description, there are many variants. For instance, the methods can be totally unsafeguarded, in which case anything goes, and they may not converge to anything, even on convex problems. Or they can be safeguarded. Safeguarded methods are usually based on trust regions or line search, and are meant to ensure convergence to something. Very importantly, just knowing that a method is L-BFGS does not by itself tell you what type of safeguarding, if any, is used. It's kind of like saying that a car is a 4-door sedan - but of course not all 4-door sedans are the same in performance or reliability. It is just one attribute of an optimization algorithm.
