I am working with ML models where the data is plentiful: I can get billions of records for training and validating my models. In fact, I have so much data, I need to sample to reduce the data set sizes (to a few tens of millions of records).

In that context, do I need to strive to do the splitting and sampling "right", i.e. splitting first then sampling with replacement for the training and validation data sets? It seems that even if I sample without replacement, then split, given the abundance of unique data points, I will be "close enough" to being i.i.d. and the risk of leaking the validation set in the training would be negligible?

(In my case there is a performance difference between sampling with and without replacement, and I would like to avoid the sampling with replacement, if safe enough)

  • 1
    $\begingroup$ For curiosity, in which proportions would you split this huge amount of data among train, test and validation? $\endgroup$ – Ale Sep 28 at 19:51
  • $\begingroup$ 80/20 probably. Tbh, in internet companies (think FB, Google, MSFT, Apple, Twitter), these sizes of data set are quite common. Gone is the time when statistics required the Student distribution because they had ~ 30 samples ... $\endgroup$ – Frank Sep 28 at 22:44
  • $\begingroup$ With such a long dataset what about using a 99/1% split? That 1% will still be made of millions of observations right? I would try splitting a validation set, then subsampling a train set accordingly. At least this is what I got in Andrew Ng course of deep learning $\endgroup$ – Ale Sep 29 at 6:17
  • $\begingroup$ @Ale - good idea. $\endgroup$ – Frank Sep 29 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.