9
$\begingroup$

When splitting up my labeled data into training, validation and test sets, I have heard everything from 50/25/25 to 85/5/10. I am sure this depends on how you are going to use your model and how prone to overfitting your learning algorithm is. Is there a way to decide or is it all by rule of thumb? Even ELSII seems vague on the subject.

$\endgroup$
13
$\begingroup$

Split sample validation without resampling (cross-validation, or better: bootstrapping) is unreliable unless you have an enormous sample (e.g., $N>20000$). Rigorous internal validation using the bootstrap is usually preferred, assuming that you program all model selection steps so they can be repeated in each bootstrap loop. And one of the problems with split sample approaches, besides volatility, is the difficulty in choosing the split fractions.

$\endgroup$
  • $\begingroup$ And what if you are working at larger scale data (but not big data) of 10000<N<1000000? At that point splitting seems reasonable. This fits many, but not all, situations I encounter. $\endgroup$ – Ed Fine Apr 2 '13 at 20:05
  • $\begingroup$ It could be quite reasonable. $\endgroup$ – Frank Harrell Apr 2 '13 at 20:07
  • $\begingroup$ I have N=95,000,000 (hold out set of 9,500,000). Where is a reference that tells me I don't have to repeat my experiment 10x? $\endgroup$ – dranxo Feb 4 '14 at 22:52
  • 2
    $\begingroup$ Just run twice (2 splits) and you'll how much results vary. They probably vary so little that you only need one split. Think of the width of a confidence interval for a proportion with such a big sample size. $\endgroup$ – Frank Harrell Feb 4 '14 at 23:58
3
$\begingroup$

Depending on the application, you could likely skip the uncertainty, and instead use bootstrapping.

Wiki: http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

Related question here. Understanding bootstrapping for validation and model selection

$\endgroup$
3
$\begingroup$

Of course you also have to decide about the splitting ratios for (double) resampling...

However, resampling usually works for quite a wide range of splitting ratios, if you keep in mind

  • not to do a leave-one-out if that would reduce the number of possible distinct runs
  • leave enough training cases in the innermost training set so the traing algorithm has a decent chance to produce a useful model.
  • the more independent cases you have, the less important are these considerations.

And what if you are working at larger scale data (but not big data) of 10000 < N < 1000000?

What you can do if you are not sure wheher resampling is need is: resample a few times. Enough so you can measure whether the resampling was necessary.

  • check the stability of your predictions
  • check the stability of your model parameters

With these results, you can decide whether you should add more resampling iterations or whether things are fine as they are.

$\endgroup$
2
$\begingroup$

There is no hard and fast rule for this. But empirical analysis showed that the more training data you have, the better your accuracy will be. But whatever you do, don't forget to put all of your training/ validation/ test data together and do a 10 fold CV when you are wrapping up. This gives a very good insight about having overfit/underfit problem during your experiment.

$\endgroup$
1
$\begingroup$

I think it all matters on which questions you are trying to answer. Are you interested in an accurate view of the performance difference between multiple algorithms? Then you need a fairly large validation set. Are you interested in how well an algorithm performs for N=10000 samples? Then you should put at least 10000 samples in the train set.

A larger validation set gives you more statistical certainty about your results, but the certainty is about the performance of an algorithm which was trained on fewer samples, which might not be what you are after in the end.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.