Validation: Data splitting into training vs. test datasets

I was naively validating my binomial logit models by testing on a test dataset. I had randomly divided the available data (~2000 rows) into training (~1500) and validation (~500) datasets.

I now read a post in another thread ( Frank Harrell) that causes me to question my approach:

Data splitting is not very reliable unless you have more than 15,000 observations. In other words, if you split the data again, accuracy indexes will vary too much from what you obtained with the first split.

How serious is this worry and what are ways around it? The OP speaks of "resampling" but not sure how that works here for validation.

Edit: Adding context as per @Bernhard's comment below:

Comparing logistic regression models

• The Frank Harrel quote is out of context. Please provide the source. Otherwise, splitting your data in the way you described seems appropriate. You may consider a split into training, test, and validation set to strengthen the validity of your model. – Bernhard Kausler Feb 22 '13 at 9:36
• Thanks @Bernhard. Sorry if it was out of context. I thought the example discussed in that thread was very similar to mine. Why do you think that was different? – curious_cat Feb 22 '13 at 11:39
• No, the context would not help. Data splitting for your sample size is not reliable. I.e. you get different model and different test data performance if you split again, and the mean squared error of your performance metric is high. – Frank Harrell Feb 22 '13 at 13:18
• You could split your data and do the testing many times: split into train and test, fit model, do diagnostics, and then repeat many times (make sure your new samples are different!) I think thats called like cross-validation or something – bdeonovic Feb 22 '13 at 13:37
• The bootstrap, as in the validate and calibrate functions in the R rms package. – Frank Harrell Feb 22 '13 at 17:14

The split sample validation you proposed above has become less popular in many fields because of the issue Harrell mentions (unreliable out of bag estimates). I know Harrell has mentioned this in his textbook, but other references would be Steyerberg "Clinical Prediction Models" p301, James et al "An Introduction to Statistical Learning" p175.

In the biomedical field boostrap resampling has thus become the standard. This is implemented in Harrell's rms package and so fairly easy to implement. But you could really use any of the other resampling methods, bootstap has just become popular because of a Steyerberg article suggesting it is the most efficient of the resampling methods ("Internal validation of predictive models: efficiency of some procedures for logistic regression analysis").

It is worth mention that the benefit of the rms package is that it easily enables you to include some of the variable selection in the bootstap (built in stepwise selection option). This can be awkward to achieve with most commercial packages.

I own sense is that the differences have been overemphasized. I usually get pretty reliable/consistent results irrespective of the method used. With large sample sizes the differences are really non-existent.

Bootstrap validation - as well as the other resampling methods - can also easily be done wrong. Often only some of the model building stages are included in the bootstrap giving inaccurate estimates. On the other hand it is fairly hard to mess up split sample validation. Given the face validity of split sampling - I know you didn't muck it up - I prefer split sample unless it is a very small dataset. It many cases the model building process is also complicated enough that it can't really be included in a resampling method.

If you want to publish in a biomedical journal though, and you aren't using a medicare size database, you will want to use a resampling method - likely bootstrapping. If the dataset is large, you can likely still get published with k-fold and save yourself some processing time.

• charles, you just attempted an edit from a different account than the one you originally posted from. Please see the help on merging the two accounts – Glen_b Nov 7 '13 at 1:22
• In my experience the main points where people mess up the validation is independent whether it is out-of-bootstrap, cross validation or set validation (unless the test set is e.g. acquired later): data-driven preprocessing (e.g. doing a PCA on the whole data, then "validating" the model built in PC score space), not splitting at the highest level of the sampling hierarchy (e.g. splitting measurements instead of patients with repeated measuerements for each patient) and data-driven optimization of hyperparameters. – cbeleites unhappy with SX Nov 7 '13 at 17:00