Are there any optimizers that perform well on small datasets? With regards to neural networks (or any optimization-based model in general), are there any optimizers that excel in the small dataset regime? Most optimizers I know of require large amounts of data to work properly.
 A: I think you confuse the role of the optimizer in neural networks. "Optimizer" is an algorithm that is intended to "optimize" parameters of a model w.r.t. some function, in this case minimize a loss function, such as mean squared error. Examples of optimizers used for training neural networks include gradient descent, stochastic gradient descent, ADAM, and others.
Optimizer may "excel" in terms


*

*how quickly it is able to find the desired optimum,

*how close it can get to the desired optimum,

*or which optimum it chooses in the usual non-convex optimization case where there are multiple optima.


However, most of these things are independent on the size of the training set.
On the other hand, there are different machine learning models (such as SVMs, neural networks, k-NN, etc.) whose performance (in terms of correctness of predictions) vary a lot with the size of the training set. Neural networks that are large enough to be able to solve difficult problems such as image segmentation have many (~millions) of parameters and require proportional amounts of training samples. In this sense, neural networks don't perform well in the small dataset regime, whereas simpler models such as logistic regression might. But this has very little to do with the optimizer.

Final remark to give an actual answer to your question, gradient descent (sometimes called batch gradient descent) is an example of an algorithm that performs better (in terms of computation speed) when the dataset is small, because it computes the gradients on the whole dataset in each iteration, as opposed to stochastic gradient descent which uses only a small part of the data in each iteration. Which of them has larger practical usability and leads to mode robust solutions is a wholly different question though.
