# Why nowadays ML algorithm rarely use optimizing functions based on newton method?

I am currently working on coursera Andrew Ng's machine learning course, and wonder why nowadays deep learning algorithms do not use fminunc (assuming a Matlab/Octave environment) for optimizing weights.

In my knowledge, fminunc does not have to set learning rate so I think it would be convenient to optimize weights compared to gradient-descent models.

• What's fminunc? – Matias Valdenegro Jul 26 '17 at 17:07
• He also used FminCG and I have combined with his neural network course and worked well – HRgiger Jul 26 '17 at 18:58
• This is very broad, especially as fminunc can be interpreted as multiple opt-approaches. But probably all are sharing the following problems: those are second-order methods -> hessian-calculation => slow and huge memory consumption (lbfgs-like methods are alternatives and sometimes used in deep-learning as the hessian is approximated using less memory); also: they don't have learning-rates because they use line-search-algorithms. These are very slow too! So basically: those algorithms don't scale well for much data! (and they might also behave different in regards to (global) results) – sascha Jul 26 '17 at 19:04

I assume by fminuc, you assume the function from Matlab or Octave. I took the liberty of editing your question to add the corresponding tags. If I do this in octave

>> help fminunc


among other things, I get this line

Function File: [X, FVAL, INFO, OUTPUT, GRAD, HESS] = fminunc (FCN,
...)


This doesn't tell me what algorithm is exactly used by this function, but there is one alarming variable that screams that this function is not fit for training neural networks: the varible HESS.

So, this function computes the Hessian. This does not surprise me since in your question, you said that it did not need a learning rate. Minimization functions that do not need a learning rate need to be, simply put, aware of the curvature of the cost function. Well known examples are Gauss-Newton and Levenberg-Marquardt in which the Hessian is explicitly computed. On the other hand, you have other, lighter algorithms like l-bfgs that only approximate the Hessian. Eitherway, this is way too expensive. Imagine your cost function is

$$$F:\mathbb{R}^n\rightarrow\mathbb{R}^m$$$

then, its Hessian is of size $n \times n$. That is ok if you have a reasonnably small function. But, in deep learning, you easily can end up with millions of parameters, i.e. $n=10^6$. So the Hessian is way to big to compute! This should also answer your question on Newton's method, since its very essence is to use the Hessian.

This is why algorithms like stochastic gradient descent are popular in machine learning. Plus, don't forget that even normal gradient descent is impossible in deep learning, because you have to approximate the gradient of the cost function using batches.

Also, one could argue that this line from the documentation

'fminunc' attempts to determine a vector X such that 'FCN (X)' is a
local minimum.


goes agains what you want in deep learning, since you seek the global minimum. However, as this paper seems to indicate (I might be wrong, I haven't read all the details), this is probably not a problem.

• Please use math formatting for mathematical expressions. More information: math.meta.stackexchange.com/questions/5020/… – Sycorax says Reinstate Monica Jul 27 '17 at 14:43
• As you can see this was migrated from stackoverflow, on which Latex is not directly available. I will update this as soon as I can , however I think that this is the responsibility of the ones who migrate the question. This just makes me look bad. – Ash Jul 27 '17 at 14:50
• Done. Also, I don't know why this was migrated, I don't see how this is a better fit for stats.stackexchange than for stackoverflow. By the logic of whoever migrated this, all my question/answers on stackoverflow, as well as many other users from the vision/robotics/machine learning community should be migrated. This question would also be perfectly welcome at robotics.stackexchange, or cross validated, etc. – Ash Jul 27 '17 at 15:00
• You're welcome. Please link me when you do create the meta thread, because I'd like to follow it. – Sycorax says Reinstate Monica Jul 27 '17 at 15:27
• @Sycorax Done. However, I am not sure I chose the best title. meta.stackoverflow.com/questions/353941/… – Ash Jul 27 '17 at 20:35

If there are a lot of parameters to optimize like in deep neural network, Hessian can be computationally expensive.

Here is Mark's comment in this question.

Is that true Newton's Method / Quasi Newton Method are not widely used in deep neutral network training?

The L, as in L-BFGS, can deal with a larger number of variables than BFGS. L stands for limited memory, and the Hessian approximation is never explicitly formed with L-BFGS., rather, some small number of the most recent gradients are retained to allow the computations needed with the Hessian approximation. But L eventually runs out of gas as well.