12
$\begingroup$

When splitting up my labeled data into training, validation and test sets, I have heard everything from 50/25/25 to 85/5/10. I am sure this depends on how you are going to use your model and how prone to overfitting your learning algorithm is. Is there a way to decide or is it all by rule of thumb? Even ELSII seems vague on the subject.

$\endgroup$
  • $\begingroup$ This Stackoverflow Q&A has two good answers on the topic with >30 upvotes each. stackoverflow.com/questions/13610074/… $\endgroup$ – Luke Singham Jul 25 '17 at 22:26
  • $\begingroup$ One thing is for sure. The proportion of training data has to be greater in all instances of model runs. $\endgroup$ – Shaurya Jauhari Jun 22 at 2:04
  • $\begingroup$ Can you expand on this to explain why the training proportion must be larger? Some prominent successes have been found using a smaller portion of training data. One is the winner of the KDD99 cup: Pfahringer "Winning the KDD99 Classification Cup: Bagging & Boosting" uses 10% of the data for training and 90% for testing. $\endgroup$ – Sycorax Jun 22 at 3:34
14
$\begingroup$

Split sample validation without resampling (cross-validation, or better: bootstrapping) is unreliable unless you have an enormous sample (e.g., $N>20000$). Rigorous internal validation using the bootstrap is usually preferred, assuming that you program all model selection steps so they can be repeated in each bootstrap loop. And one of the problems with split sample approaches, besides volatility, is the difficulty in choosing the split fractions.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ And what if you are working at larger scale data (but not big data) of 10000<N<1000000? At that point splitting seems reasonable. This fits many, but not all, situations I encounter. $\endgroup$ – Ed Fine Apr 2 '13 at 20:05
  • $\begingroup$ It could be quite reasonable. $\endgroup$ – Frank Harrell Apr 2 '13 at 20:07
  • $\begingroup$ I have N=95,000,000 (hold out set of 9,500,000). Where is a reference that tells me I don't have to repeat my experiment 10x? $\endgroup$ – dranxo Feb 4 '14 at 22:52
  • 2
    $\begingroup$ Just run twice (2 splits) and you'll how much results vary. They probably vary so little that you only need one split. Think of the width of a confidence interval for a proportion with such a big sample size. $\endgroup$ – Frank Harrell Feb 4 '14 at 23:58
3
$\begingroup$

Depending on the application, you could likely skip the uncertainty, and instead use bootstrapping.

Wiki: http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

Related question here. Understanding bootstrapping for validation and model selection

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

Of course you also have to decide about the splitting ratios for (double) resampling...

However, resampling usually works for quite a wide range of splitting ratios, if you keep in mind

  • not to do a leave-one-out if that would reduce the number of possible distinct runs
  • leave enough training cases in the innermost training set so the traing algorithm has a decent chance to produce a useful model.
  • the more independent cases you have, the less important are these considerations.

And what if you are working at larger scale data (but not big data) of 10000 < N < 1000000?

What you can do if you are not sure wheher resampling is need is: resample a few times. Enough so you can measure whether the resampling was necessary.

  • check the stability of your predictions
  • check the stability of your model parameters

With these results, you can decide whether you should add more resampling iterations or whether things are fine as they are.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

There is no hard and fast rule for this. But empirical analysis showed that the more training data you have, the better your accuracy will be. But whatever you do, don't forget to put all of your training/ validation/ test data together and do a 10 fold CV when you are wrapping up. This gives a very good insight about having overfit/underfit problem during your experiment.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I think it all matters on which questions you are trying to answer. Are you interested in an accurate view of the performance difference between multiple algorithms? Then you need a fairly large validation set. Are you interested in how well an algorithm performs for N=10000 samples? Then you should put at least 10000 samples in the train set.

A larger validation set gives you more statistical certainty about your results, but the certainty is about the performance of an algorithm which was trained on fewer samples, which might not be what you are after in the end.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.