I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why is 2nd order derivative optimization methods better for NN without hidden layers?
$\begingroup$
$\endgroup$
5
-
1$\begingroup$ The observation "That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better" may not be universally correct. Are there any authoritative publication showing this? $\endgroup$– patagonicusCommented Oct 20, 2021 at 22:02
-
$\begingroup$ @MehmetSüzen not that I know of. The argument in the blog is that regressions are not trained using SGD and 2nd order derivative methods therefore since a simple regression is most similar to a non-hidden layered NN that it's probably better to use this 2nd order derivative method $\endgroup$– user8714896Commented Oct 20, 2021 at 22:08
-
$\begingroup$ Interesting, this could be an open research problem. $\endgroup$– patagonicusCommented Oct 20, 2021 at 22:37
-
2$\begingroup$ There is a good study on this topic by Tom Minka tminka.github.io/papers/logreg/minka-logreg.pdf (spoiler: conjugate gradient descent and quasi-Newton algorithms work well). I suspect simply second order methods (e.g. Iteratively Reweighted Least Squares) are only good for rather small models. $\endgroup$– Dikran MarsupialCommented Oct 24, 2021 at 14:30
-
1$\begingroup$ I suspect for large models with very large datasets, it may be better to start off with SGD to get the weights into the ballpark and then polish them with SCGD, because you don't need all of the data to find the ballpark. Just GD rather than SGD is likely to get you to a good solution fairly quickly, especially if you use something like RPROP. $\endgroup$– Dikran MarsupialCommented Oct 24, 2021 at 14:52
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The optimization task in the blog post, a classification task with cross-entropy loss, is convex when there are no hidden layers, so you might expect both first and second order optimization methods to be able to converge arbitrarily well.
However, you would expect second order methods to converge in fewer iterations -- for example, a second order method can locate the min/max of a quadratic function with just one step. On the other hand, a first order method needs many iterations, but this is balanced by the fact that computing the first derivative is much cheaper than computing the hessian, especially when the dimensionality becomes big. The author didn't seem to take this into account, and ran both methods for the same number of iterations.
Another factor might be that L-BFGS implementations are often written with convergence as a goal, whereas no one really expects NN optimizers like Adam to "converge" -- when training a deep neural network, convergence to a local minimum isn't the goal, so the default parameters of Adam are probably not tuned accordingly -- for example, for gradient descent to converge, it's usually necessary to decay the step-sizes to 0, but this not the default behavior in most machine learning libraries.
-
2$\begingroup$ I suspect it is the "stochastic" in SGD that is preventing it from converging to the exact minima as the updates are noisy as they depend on the batch used in each iteration. I don't think it is a clean test of first-order versus second order optimisation. $\endgroup$ Commented Oct 24, 2021 at 14:55
-