0
$\begingroup$

I have split my training data into 5 sets. I am using a basic linear model with all of my predictive variables (because I only have a handful). I repeatedly, manually, set up 5 linear models that train-on-4-of-the-sets, test-on-the-5th-set. I grabbed the RMSE and the Accuracy Percentage for prediction within each of the 5 models.
My question is - what do I do with this information? If I run this exact model on the full 70%training set, then run the predictions on the original 30%test set in order to determine my accuracy, what is the point of the cross validation?

Note: I am running many, many, many models- BoxCox, RandomForest, etc, so I need to be able to compare the prediction abilities of the models- hence the reason that I feel as though I need to run every model again the original 30%test set- for sake of comparison.

Example in R:

#manual cross validation
CV1 <- lm(y1 ~ x1+x2+x3+x4, data=TrainData1)
CV1Predictions <- predict(CV1, TestData1)
mean(CV1Predictions==TestData1$y1)   #accuracy measure

CV2 <- lm(y1 ~ x1+x2+x3+x4, data=TrainData2)
CV2Predictions <- predict(CV2, TestData2)
mean(CV2Predictions==TestData2$y1)
.
.
. #this continues for 5 sets
$\endgroup$
0
$\begingroup$

You are describing (1) split sample validation (2) cross-validation K=5. Both are forms of internal validation with their own strengths and weaknesses. In particular, split sample validation - (30%) test set - can be unstable with small sample sizes.

EDIT
(1) for the lm model above, there is no utility in doing both forms of internal validation. Just choose one, realizing that either form of internal validation has its own limitations.
(2) For machine learning algorithms with hyper-parameters, both forms of validation might be used. So in your example cross-validation may be used to choose mtry value for the randomForest model and then split sample validation could be used to estimate model accuracy. The cross-validation and split-sample are being used with different objectives in mind.

$\endgroup$
2
  • $\begingroup$ If I need to run the same model on the 30%test set anyways, what is the purpose of the cross validation? $\endgroup$ – Nameless Mar 14 '16 at 16:04
  • $\begingroup$ Data splitting only works with very large sample sizes, and cross-validation must be repeated up to 100 times and averaged for its volatility to be controlled. $\endgroup$ – Frank Harrell Feb 5 '19 at 12:25
0
$\begingroup$

The point of cross-validation: Cross-validation (CV) is used when your data set is too small to perform both training and testing. Following your example, if you need 80% of your data to train, then the test set might be too small to properly determine accuracy. In other words, the test set is too small to be representative for all your outcomes. Or the other way around: Say you need 50% percent of your data to properly determine accuracy, and the 50% left for training results in a very unstable model. So, you need 80% for training and 50% for testing, and you have a problem. CV will solve this problem for you. For each fold you get n/K predictors $\hat{y}_i$'s, and after running trhough the $K$ folds you evaluate the $n$ predictors as you would if they were a test set.

Note: There are a lot, a lot of pitfalls when performing CV. So if you do have enough data, as you suggest, a separate test set is better. This book (available for free online) https://web.stanford.edu/~hastie/ElemStatLearn/ gives a good introduction to CV and its pitfalls. I strongly recommend against CV unless it is absolutely necessary and you have a good understanding of its mechanisms.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.