Why is a 2nd order derivative optimization better for no hidden layer neural networks? I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why is 2nd order derivative optimization methods better for NN without hidden layers?
 A: The optimization task in the blog post, a classification task with cross-entropy loss, is convex when there are no hidden layers, so you might expect both first and second order optimization methods to be able to converge arbitrarily well.
However, you would expect second order methods to converge in fewer iterations -- for example, a second order method can locate the min/max of a quadratic function with just one step. On the other hand, a first order method needs many iterations, but this is balanced by the fact that computing the first derivative is much cheaper than computing the hessian, especially when the dimensionality becomes big. The author didn't seem to take this into account, and ran both methods for the same number of iterations.
Another factor might be that L-BFGS implementations are often written with convergence as a goal, whereas no one really expects NN optimizers like Adam to "converge" -- when training a deep neural network, convergence to a local minimum isn't the goal, so the default parameters of Adam are probably not tuned accordingly -- for example, for gradient descent to converge, it's usually necessary to decay the step-sizes to 0, but this not the default behavior in most machine learning libraries.
