Does dropout have any benefits when overfitting isn't a concern? I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training dataset to overfit on. I've confirmed experimentally that performance on a fixed validation dataset is no worse than training performance.
Is there any potential/likely benefit to including dropout in this model? Is there anything to support it resulting in better trained performance when overfitting isn't a concern?
I've tried some basic experiments and found that dropout leads to dramatically reduced performance in the early stages of training, but I haven't let it run for very long.
 A: Even if overfitting is not a concern, dropout could still help. Neural networks are (most of the time) trained with backpropagation, i.e. stochastic gradient descent (SGD). And it is known that the stochastic aspect of SGD often helps it to escape local minima. While the main source of stochasticity is the random blocks in each epoch (with the blocks much smaller than the total dataset), techniques like dropout (might) add a different type of stochasticity that could perhaps improve your results.
So, while you can never be certain of anything with DNNs, I would definitely give dropout a try.
A: One interpretation of dropout is that you can analog it as an ensemble of many neural nets.
When training, each node in the networks has some probability to be dropped.
This means that in every training step, the network architectures are different.
And when testing, we weightedl combined many networks together into a single one.
However, in training, the dropped networks' capabiliteis are less capatible then the full networks.
Thus the performance of the ensembled one might still not outperforme the full one without dropout in training.
