# Why nowadays ML algorithm rarely use optimizing functions based on newton method?

I am currently working on coursera Andrew Ng's machine learning course, and wonder why nowadays deep learning algorithms do not use fminunc (assuming a Matlab/Octave environment) for optimizing weights.

In my knowledge, fminunc does not have to set learning rate so I think it would be convenient to optimize weights compared to gradient-descent models.

• What's fminunc? Jul 26, 2017 at 17:07
• He also used FminCG and I have combined with his neural network course and worked well
– HRgiger
Jul 26, 2017 at 18:58
• This is very broad, especially as fminunc can be interpreted as multiple opt-approaches. But probably all are sharing the following problems: those are second-order methods -> hessian-calculation => slow and huge memory consumption (lbfgs-like methods are alternatives and sometimes used in deep-learning as the hessian is approximated using less memory); also: they don't have learning-rates because they use line-search-algorithms. These are very slow too! So basically: those algorithms don't scale well for much data! (and they might also behave different in regards to (global) results) Jul 26, 2017 at 19:04

I assume by fminuc, you assume the function from Matlab or Octave. I took the liberty of editing your question to add the corresponding tags. If I do this in octave

>> help fminunc


among other things, I get this line

Function File: [X, FVAL, INFO, OUTPUT, GRAD, HESS] = fminunc (FCN,
...)


This doesn't tell me what algorithm is exactly used by this function, but there is one alarming variable that screams that this function is not fit for training neural networks: the varible HESS.

So, this function computes the Hessian. This does not surprise me since in your question, you said that it did not need a learning rate. Minimization functions that do not need a learning rate need to be, simply put, aware of the curvature of the cost function. Well known examples are Gauss-Newton and Levenberg-Marquardt in which the Hessian is explicitly computed. On the other hand, you have other, lighter algorithms like l-bfgs that only approximate the Hessian. Eitherway, this is way too expensive. Imagine your cost function is

$$$F:\mathbb{R}^n\rightarrow\mathbb{R}^m$$$

then, its Hessian is of size $n \times n$. That is ok if you have a reasonnably small function. But, in deep learning, you easily can end up with millions of parameters, i.e. $n=10^6$. So the Hessian is way to big to compute! This should also answer your question on Newton's method, since its very essence is to use the Hessian.

This is why algorithms like stochastic gradient descent are popular in machine learning. Plus, don't forget that even normal gradient descent is impossible in deep learning, because you have to approximate the gradient of the cost function using batches.

Also, one could argue that this line from the documentation

'fminunc' attempts to determine a vector X such that 'FCN (X)' is a
local minimum.


goes agains what you want in deep learning, since you seek the global minimum. However, as this paper seems to indicate (I might be wrong, I haven't read all the details), this is probably not a problem.

– Sycorax
Jul 27, 2017 at 14:43
• As you can see this was migrated from stackoverflow, on which Latex is not directly available. I will update this as soon as I can , however I think that this is the responsibility of the ones who migrate the question. This just makes me look bad.
– Ash
Jul 27, 2017 at 14:50
• Done. Also, I don't know why this was migrated, I don't see how this is a better fit for stats.stackexchange than for stackoverflow. By the logic of whoever migrated this, all my question/answers on stackoverflow, as well as many other users from the vision/robotics/machine learning community should be migrated. This question would also be perfectly welcome at robotics.stackexchange, or cross validated, etc.
– Ash
Jul 27, 2017 at 15:00