Many claim different proportion between test and train set in regression and classification telling this should be test:train : 0.25:0.75 or 0:33:0:66 (the most popular with what I met). But what with proportions 0.1:0.9 or even 0.05:0.95? I am asking because there are over 1,2 mln observations in a set I'd like to analyze and 0.05 from 1,2 mln is a huge number itself (60000).

What do you think about this?


Classification requires a much larger sample size than prediction if classification involves categorizing $Y$ so beware.

There is no uniquely good answer to your question. One way of thinking about it is that you need the test sample to be large enough so that the entire calibration curve can be estimated with excellent precision. Then see if what's left is large enough for a training sample that is likely to yield a reliable model fit. There are many reasons to use the optimism bootstrap instead. With it there is only one choice -- the number of bootstrap repetitions (400 is usually large enough; for your sample size fewer are OK). The bootstrap yields higher precision of estimates of predictive accuracy than data splitting or 10-fold cross-validation.

For more information see http://biostat.mc.vanderbilt.edu/RmS#Materials

  • $\begingroup$ Thanks for quite rich answer. Could you maybe post some references? $\endgroup$ – Marcin Kosiński Nov 29 '14 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.