1
$\begingroup$

I am training an LSTM neural network for nlm on a big dataset: the model has about 100M learnable parameters and the dataset consists of about 2G characters.

Therefore it seems that overfitting should not be a problem. Should I still try to apply dropout to my model? How should I determine this? (would you please link something theoretically grounded if possible).

$\endgroup$
4
$\begingroup$

Pragmatically, I would train first without dropout and check validation loss to see if there is overfitting. If there isn't, then there is no problem. If there is, then go back and add dropout.

I don't think you can determine if there will be overfitting before hand. It's important to keep in mind that you can overfit even if the number of parameters is much less than the size of the data. This is probably the case in most image-segmentation tasks, where there are on the order of 1 million labels per image and only several million parameters in the network, yet it's still possible to overfit here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.