Train a model on batches with multiple epochs vs. each batch with multiple epochs In traditional supervised learning, say we split the data into $B$ batches, we will go through all these batches with $E$ epochs. The computational cost is $E\times B$.
But what if on each batch, we train the model for $E$ epochs, and then fetch the next batch till we go through all the batches. The computational cost remains the same. I tried it empirically on a neural network, the first method is better than the second one. But will it cause any difference mathematically?
 A: This idea was discussed on r/MachineLearning here a few years ago. It was pointed out there that Fischetti et al. (2018, unpublished) proposed doing exactly this, and found good results for one very limited setting, training AlexNet on CIFAR-10. Other commenters said that it performed poorly for them.
Some thoughts:

*

*If you use a very low learning rate, this is basically the same as multiplying the learning rate by $E$. But this is not true for reasonably-large learning rates.

*If you’re using Adam / other adaptive optimizers, if you’re using momentum, or if you’re using batch norm / etc, you’re really breaking the iid assumption and things could maybe go a little wonky.

*If you would have trained your minibatch algorithm for 100 epochs, you probably don’t want to repeat each batch 100 times. Probably better to repeat each batch 2-5 times, and then run through that whole process (re-shuffling the batches) 20-50 times.

*As a pure optimization problem for the training error, I think (but am not 100% sure) that theoretically, this method should converge to low training error in roughly the same situations as the standard minibatch SGD, and perhaps not much slower. I’m not really an optimization expert, though, this is entirely a guess.

*It is likely that, at least in some settings, it will converge to a different training-error solution than standard SGD. (The “implicit regularization” is different.) I’m not aware of anyone having studied this theoretically; it seems unlikely, since mathematical understanding of the implicit bias of the standard algorithm people actually use is still pretty limited (for deep networks) at this point.

*It might encourage overfitting to each training batch more than standard SGD, which could be bad.

Overall, though, I don’t think anyone’s really mathematically studied this setting. In “usual” single-machine training regimes, if you do data loading in parallel and pin CUDA memory / etc, loading the data isn’t really the computational bottleneck, and so there doesn’t seem to be a ton of reason to do this. In some regimes, though, it might be faster to run, and so could be worth exploring with a fairly small amount of “persistence.”
