0
$\begingroup$

I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how well these models predict new observations by using 5-fold Cross Validation. From my understanding 5-fold Cross Validation shuffles then splits my data into 5 groups and chooses 1 for the testing set, and the other 4 for the training set. A given model is tested and the prediction error is recorded. This is repeated until all 5 groups are used as a testing set. Finally, the prediction errors are averaged.

My question is, when I am first determining the multiple regression equation (checking for assumptions, applying transformations, variable selection, etc.) which set of data should I use as my training set? Do I use the entire data set? Do I use one of the 5 training sets created by the 5-folds CV method? Do I repeatedly try to fit the regression model for all 5 training sets? If so, how would I extract each training set using the caret package?

$\endgroup$
1
$\begingroup$

This blog talks about models being "y-aware". Essentially, anytime you use the outcomes to make a decision about the model, then that data can not be used in subsequent steps of model selection/development. Because the process you describe is essentially a form of hyperparameter optimization, then your model selection process is y-aware.

Therefore, your best option is to actually do the selection procedure within the cross-validation as a sort of nested cross validation. Here is a link to a blogpost by caret's author which outlines how this procedure can be done.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.