When do we train a neural network using full batch training?

Question

It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that it is possible to train the network on one datapoint at the time. It is quite rare, on the other hand, to see research papers, or state of the art models, that are trained on a whole dataset simultaneously, i.e., in full-batch training.

For a research project, it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches. An example I have found is the COIN and COIN++ papers, that train on full batch training in order to memorise datapoints.

Question:

Do you know other applications where performing full-batch training is preferable to use mini-batches? Which ones?

Answer 1

To be honest the COIN paper is an exceptional case. The dataset consists of only 24 images, which is smaller than even the minibatch size used in more traditional experiments. People like to use mini-batches because they introduce noise which helps with generalization, which is often what the researcher is interested in. However, even if the research does not care for that, most of the time the datasets are too big to even fit at once into the memory of the GPU.

I guess in all scenarios where overfitting is desired, like compression, it might make sense to use full batch training. However, as soon as you want to compress a large number of images the COIN approach will probably fail. As you cannot load them into the memory, and you would need to train multiple networks. IMO

So in general I could only think about cases where the implicit neural representation is needed, it might make sense to use full-batch training