This may be a silly idea. I have a huge dataset with billions of data points, and it would take a long time to run an epoch over all the data set. So
I was thinking of using the following strategy:
- Train a deep NN on a small sample of the data, constantly saving checkpoint models, and stop it when validation loss plateaus.
- Take the best model from step 1, append new data to the training set, and train again until validation loss plateaus.
- Repeat steps 1 and 2.
My intuition is as follows: perhaps I don't need to use all the data to train a model. Therefore, at every point, the model learns what it can from the small dataset (without overfitting), and once it starts to overfit we simply give it mor data. This strategy would ensure that the model takes only the minimum amount of data needed to ensure a good model, and would decrease training time.
Is this a coherent strategy? Would this be a faster way of training, or would it introduce some variance?