1
$\begingroup$

It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that it is possible to train the network on one datapoint at the time. It is quite rare, on the other hand, to see research papers, or state of the art models, that are trained on a whole dataset simultaneously, i.e., in full-batch training.

For a research project, it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches. An example I have found is the COIN and COIN++ papers, that train on full batch training in order to memorise datapoints.

Question:

Do you know other applications where performing full-batch training is preferable to use mini-batches? Which ones?

$\endgroup$
1
  • 1
    $\begingroup$ One practical problem is that neural networks usually need a lot of data, so in many cases with full-batch training, you would run out of memory very fast. $\endgroup$
    – Tim
    Commented Oct 10, 2022 at 11:57

1 Answer 1

2
$\begingroup$

To be honest the COIN paper is an exceptional case. The dataset consists of only 24 images, which is smaller than even the minibatch size used in more traditional experiments. People like to use mini-batches because they introduce noise which helps with generalization, which is often what the researcher is interested in. However, even if the research does not care for that, most of the time the datasets are too big to even fit at once into the memory of the GPU.

I guess in all scenarios where overfitting is desired, like compression, it might make sense to use full batch training. However, as soon as you want to compress a large number of images the COIN approach will probably fail. As you cannot load them into the memory, and you would need to train multiple networks. IMO

So in general I could only think about cases where the implicit neural representation is needed, it might make sense to use full-batch training

$\endgroup$
1
  • $\begingroup$ Thank you for the answer! Yes, I'm aware that COIN is an exceptional case, I was simply wondering whether there are other specific and exceptional cases in the literature similar to this one. $\endgroup$
    – Alfred
    Commented Oct 11, 2022 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.