Any theory on how to split the data?

Question

Splitting data for learning and evaluation is a pretty common practice. I've seen people do (train, test, dev) = (50%, 25%, 25%) or (50%, 30%, 20%), etc. Clearly one point is to have enough data for learning purpose, and at the same time, bigger test/dev will give a more significant evaluation. Is there any theory on how to split the data to "train", "test", "dev"? Why do we split this way? Why no one does (33%, 33%, 33%)?

Answer 1

This may not be quite what you asked, but one major theoretical point of caution when splitting the data is that you shouldn't put a set of correlated observations (i.e. correlated even after conditioning on your features) into both training and the test set. You need to take some stand on what in some sense is an independent observations (or set of observations). For example, imagine:

• each record/observation is a medical test result.
• you have multiple test results per person.
• the test results for each person are correlated through unobserved, individual specific characteristics.

When constructing a training set and a test set, you could split:

1. record wise: randomly assign each record to training or test set
2. subject wise: randomly assign people (and all their test results) to training or test set

Record wise splitting into training and test sets is in some sense is like running validation on your training data! And it may give horribly biased validation results. In this case, you would want to do (2).

Answer 2

Summary: there are known guidelines (at least for some situations) for required sample sizes - but IMHO they cannot easily be converted into a "split into such-and-such fractions" rule of thumb.

---

While in k-fold cross validation the choice of k often doesn't matter much there are additional coniderations for single splits.

The reason why the k in k-fold cross validation doesn't have much influence (outside extreme choices like $k = n$) is that at the end of the cross validation run each case has been tested once. The difference is only how many surrogate models you compute in order to achieve this.

For single splits, the following considerations can help, though you'll notice that they are not easily expressed as fraction of total samples available.

• Training data: necessary training sample sizes are often conveniently discussed as training cases relative to the needed/desired model complexity.
• In contrast, the uncertainty on test results due to the limited number of test cases depends on the absolute number of test cases. In general this is true for both regression and classification, however classification 0/1 loss and the related proportions (figures of merit such as accuracy, true positive rates, predictive values etc.) have a particularly nasty behaviour in this respect.

• However, the upside of this is that you can calculate necessary absolute test sample sizes for the testing of the final model for different scenarios such as "the 95% confidence interval of the final sensitivity estimate should not be wider than 10 percentage points".

• Data-driven model optimization/selection: from a statistical point of view, this will often require much larger test sample sizes than the final evaluation in order to avoid "skimming testing variance" i.e. spurious (accidental) optimization "results" (which I'd consider failure of the optimization - however, this type of failure is not routinely detected/warned against). Also, the necessary optimization testing sample size depends crucially on the number of comparisons to be made.
  All in all, I'd expect that the optimization sets should typically be larger than the final evaluation sets - or that instead of data-driven optimization, reasonable hyperparameters may be fixed using external knowledge (the sample sizes I encounter typically do not allow optimization driven by test results)

---

For classification 0/1 loss and proportion-type figures of merit, we put these thoughts into a paper:
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33. DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

Answer 3

I don't have a theory based response, but my allocation of data usually depends on how much data I have and how well my model fits that data. If you have a bunch of data, you can probably be pretty liberal in how much you set aside for validation and testing. If you are limited, that is where model fit comes in. Imagine you are making a linear regression model where your outcome y is perfectly dependent on you feature x with absolutely no noise in the system. In this case, you would never need more than 2 data points to get a perfect fit, so even in a small data set, you can still leave out a bunch for validating and testing. On the other hand, if there is a lot of noise in the system, you may have to devote more data as a training set in order to find the signal.

One way to see which situation you are in is to start with a smaller training set and boot strap it to produce a few models. Then take a look at your model coefficients in the various rounds of boot strapping. If your coefficients are pretty stable, then chances are you gave the model enough data to find what ever signal there is to find. If they bounce around a lot, then the model probably needs more data. Keep in mind, you are not looking at errors of predictions here, just the stability of the over all model. You can then verify the model stability by either adding a little more data to your training set and repeating your boot strapping or you can select a new subset of your data of the same size and repeat to make sure that you didn't just pick weird data the first time.

Answer 4

To add to other comments I think there is no rule gainst seeing this percentage as a hyper(or maybe better to say hyper-hyper) parameter. The set up that I can think of is dividing your data by two at first. Let's say put 10% of your data aside (yeah ! it's percentage again) and then for the rest of your data slice it in different ways (30,30,30), (80,10,10). In each scenario, you'll get training, validation and test rates & you should be able to compare it to the rate that you get on that 10% test set and see if you can find any constant improvement because of a specific scenario. Like some other parts of machine learning, we may test it first and then try to find a reason for what we see (finding a relationship between the amount of data and these values, if there is any!