When splitting up my labeled data into training, validation and test sets, I have heard everything from 50/25/25 to 85/5/10. I am sure this depends on how you are going to use your model and how prone to overfitting your learning algorithm is. Is there a way to decide or is it all by rule of thumb? Even ELSII seems vague on the subject.
Split sample validation without resampling (cross-validation, or better: bootstrapping) is unreliable unless you have an enormous sample (e.g., $N>20000$). Rigorous internal validation using the bootstrap is usually preferred, assuming that you program all model selection steps so they can be repeated in each bootstrap loop. And one of the problems with split sample approaches, besides volatility, is the difficulty in choosing the split fractions.
Depending on the application, you could likely skip the uncertainty, and instead use bootstrapping.
Related question here. Understanding bootstrapping for validation and model selection
Of course you also have to decide about the splitting ratios for (double) resampling...
However, resampling usually works for quite a wide range of splitting ratios, if you keep in mind
- not to do a leave-one-out if that would reduce the number of possible distinct runs
- leave enough training cases in the innermost training set so the traing algorithm has a decent chance to produce a useful model.
- the more independent cases you have, the less important are these considerations.
And what if you are working at larger scale data (but not big data) of 10000 < N < 1000000?
What you can do if you are not sure wheher resampling is need is: resample a few times. Enough so you can measure whether the resampling was necessary.
- check the stability of your predictions
- check the stability of your model parameters
With these results, you can decide whether you should add more resampling iterations or whether things are fine as they are.
There is no hard and fast rule for this. But empirical analysis showed that the more training data you have, the better your accuracy will be. But whatever you do, don't forget to put all of your training/ validation/ test data together and do a 10 fold CV when you are wrapping up. This gives a very good insight about having overfit/underfit problem during your experiment.
I think it all matters on which questions you are trying to answer. Are you interested in an accurate view of the performance difference between multiple algorithms? Then you need a fairly large validation set. Are you interested in how well an algorithm performs for N=10000 samples? Then you should put at least 10000 samples in the train set.
A larger validation set gives you more statistical certainty about your results, but the certainty is about the performance of an algorithm which was trained on fewer samples, which might not be what you are after in the end.