I have split my training data into 5 sets. I am using a basic linear model with all of my predictive variables (because I only have a handful). I repeatedly, manually, set up 5 linear models that train-on-4-of-the-sets, test-on-the-5th-set. I grabbed the RMSE and the Accuracy Percentage for prediction within each of the 5 models.
My question is - what do I do with this information? If I run this exact model on the full 70%training set, then run the predictions on the original 30%test set in order to determine my accuracy, what is the point of the cross validation?

Note: I am running many, many, many models- BoxCox, RandomForest, etc, so I need to be able to compare the prediction abilities of the models- hence the reason that I feel as though I need to run every model again the original 30%test set- for sake of comparison.

Example in R:

#manual cross validation
CV1 <- lm(y1 ~ x1+x2+x3+x4, data=TrainData1)
CV1Predictions <- predict(CV1, TestData1)
mean(CV1Predictions==TestData1$y1)   #accuracy measure

CV2 <- lm(y1 ~ x1+x2+x3+x4, data=TrainData2)
CV2Predictions <- predict(CV2, TestData2)
. #this continues for 5 sets

You are describing (1) split sample validation (2) cross-validation K=5. Both are forms of internal validation with their own strengths and weaknesses. In particular, split sample validation - (30%) test set - can be unstable with small sample sizes.

(1) for the lm model above, there is no utility in doing both forms of internal validation. Just choose one, realizing that either form of internal validation has its own limitations.
(2) For machine learning algorithms with hyper-parameters, both forms of validation might be used. So in your example cross-validation may be used to choose mtry value for the randomForest model and then split sample validation could be used to estimate model accuracy. The cross-validation and split-sample are being used with different objectives in mind.

  • $\begingroup$ If I need to run the same model on the 30%test set anyways, what is the purpose of the cross validation? $\endgroup$ – Nameless Mar 14 '16 at 16:04
  • $\begingroup$ Data splitting only works with very large sample sizes, and cross-validation must be repeated up to 100 times and averaged for its volatility to be controlled. $\endgroup$ – Frank Harrell Feb 5 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.