4
$\begingroup$

i have a Dataset of 5000 samples for a regression problem, now with this number of samples can i and is it better to use K-Fold cross validation instead of the validation set as an alternative?

if not i've been splitting data as 70% 15% 15%, but it doesn't seem the distribution of splits are exactly the same, i mean depending on the split i would get very different predictions, for example in split i tune the model with validation set and get 0.75 $R^2$ for validation set and then get a $R^2$ if 0.88 for test set, and in another instance its the exact opposite.

if i do a few random splits and find a few splits that the model structure is not very different in each split, can i just use one of these splits? wouldn't that defeat the purpose of the randomness of splits? out of these splits i chose two random splits, now after training, tuning and all the other steps, for one of the splits i would get the following $R^2$ for train, dev and test set: 0.91, 0.92, 0.90 and for the other one i would get these: 0.93, 0.88, 0.85. can i use the first one that gave me the best result? is it valid? i mean depending on the split it seems my final test score would be totally different.

$\endgroup$
  • 1
    $\begingroup$ What kind of data is it? i.e. time series or something with autocorrelation, or is the data more independent? If the data is not independent then you would run the risk of overtraining the model and/or coming up with a false sense of high performance. Separate validation sets are likely more rigorous a test, depending on the application. $\endgroup$ – Rob Jul 10 '18 at 15:14
  • $\begingroup$ no the data is more independent, of course some features have high correlation with each other but noting abnormal. i understand that about separate validation sets but my problem is that depending on the split ratio i would get very different predictions and i'm wondering what is the best way to minimize this problem? $\endgroup$ – john d Jul 10 '18 at 15:50
  • 1
    $\begingroup$ I think 75% 15% 15% or similar (60 20 20) makes sense. I would try a few random split sets in that ratio and see, if the model structure is very different each time it may indicate that you either don't have enough data or somehow your model structure is not capturing the most important relationships, which could be because it varies within the data. I would spend some time thinking about the regression model itself and how it is structured. Otherwise consider other approaches (conditional regression, neural networks, etc.) $\endgroup$ – Rob Jul 11 '18 at 12:44
  • $\begingroup$ i did do your suggestion and found a few splits that the model structure is not very different, but can i just use one of these splits? wouldn't that defeat the purpose of the randomness of splits? $\endgroup$ – john d Jul 11 '18 at 17:54
5
$\begingroup$

Your experience with split-dependent differences in your modeling and performance estimates is a reason why cross validation is often preferred in all but extremely large data sets. Repeated cross validation or bootstrapping might be even better.

With the separate training/validation/test approach, your estimate of the generalizability of your model's performance on new data comes from the test set, which is set aside until initial training and tuning are done on the training and validation sets. Say that you want to estimate the error in model predictions made on your test set. You are then trying to estimate a variance, which can require a surprisingly large number of cases to estimate precisely.

But if you set aside more cases for the test set so that you have a better measure of generalizability, you have fewer cases available for training and validation, which may limit your ability to generate a useful model in the first place.

The chapter on cross validation and related methods in The Elements of Statistical Learning (2nd edition, p. 222) puts it like this:

The methods in this chapter are designed for situations where there is insufficient data to split it into three parts. Again it is too difficult to give a general rule on how much training data is enough; among other things, this depends on the signal-to-noise ratio of the underlying function, and the complexity of the models being fit to the data.

So cross validation is a useful approach in cases where you don't have a "large enough" data set to accomplish your goals. Your question suggests that you might be in such a situation despite having 5000 cases.

In practice, a single run of cross validation can give imprecise results. Frank Harrell recommends repeated runs of cross validation or, better, bootstrapping to take advantage of all the data most efficiently in building and evaluating a model. See for example this answer, with a link to further reading in a comment. His rms package provides tools for building, validating, and calibrating models.

$\endgroup$
  • $\begingroup$ thank you for your comprehensive answer, i would really appreciate it if you can answer me this too. say i did two random splits for train, dev and test set, now after training, tuning and all the other steps, for one of the splits i would get the following $r^2$s for train, dev and test set: 0.91, 0.92, 0.90 and for the other one i would get these: 0.93, 0.88, 0.85. can i use the first one that gave me the best result? is it valid? i mean depending on the split it seems my final test score would be totally different. $\endgroup$ – john d Jul 12 '18 at 12:58
  • 1
    $\begingroup$ @johnd never depend on an apparent "best result" as being valid, even if it happens to be the first result. If you want your model to apply to the population as a whole, and particularly if you want to make reliable predictions on new cases, you need to guard against "best results" that depend on your particular data sample or a particular choice of data split. Repeated or nested cross validation will be more reliable. Also, you might want to consider mean-square residuals instead of $R^2$ as your measure of goodness of fit as it more directly shows how well the model fits individual cases. $\endgroup$ – EdM Jul 12 '18 at 14:40
2
$\begingroup$

In addition to the accepted answer, which is right, I would like to point that you don't need to choose between k-fold cross validation and training, validation and test set, because you can have both with nested cross validation.

In short:

  • You split your data in several test sets (just like in k-fold cross validation).
  • After taking aside each test set, you optimize your model using k-fold cross validation with the remaining data.
  • You test each optimized model against the each test set.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.