I have split my training data into 5 sets. I am using a basic linear model with all of my predictive variables (because I only have a handful). I repeatedly, manually, set up 5 linear models that train-on-4-of-the-sets, test-on-the-5th-set. I grabbed the RMSE and the Accuracy Percentage for prediction within each of the 5 models.
My question is - what do I do with this information? If I run this exact model on the full 70%training set, then run the predictions on the original 30%test set in order to determine my accuracy, what is the point of the cross validation?
Note: I am running many, many, many models- BoxCox, RandomForest, etc, so I need to be able to compare the prediction abilities of the models- hence the reason that I feel as though I need to run every model again the original 30%test set- for sake of comparison.
Example in R:
#manual cross validation CV1 <- lm(y1 ~ x1+x2+x3+x4, data=TrainData1) CV1Predictions <- predict(CV1, TestData1) mean(CV1Predictions==TestData1$y1) #accuracy measure CV2 <- lm(y1 ~ x1+x2+x3+x4, data=TrainData2) CV2Predictions <- predict(CV2, TestData2) mean(CV2Predictions==TestData2$y1) . . . #this continues for 5 sets