I have a question regarding the process of feature selection, model building and k fold cross validation.

I have a forty features and 200 records data sets. I want to select down to 10-12 features and build a model maybe linear regression. Here is my understand of the whole process.

  1. In the beginning use all the data, calculate the correlation matrix to find large pair wise correlation and drop some feature highly correlated features
  2. Use all your data, build different models and use certain measures like R square or Root mean square error to evaluate among different models
  3. Split you data and do a k fold cross validation to verify if there is over-fitting in your model to evaluate your model. Average the error from each cv as the overall error, compare different model with this overall error.
  4. Select the model

    5. Refit the model with all the data.

But I feel there is some flaw, can someone point out for me. In step 2, my model see all the data. Should I use K-fold CV from beginning? But K fold CV is for model evaluation. Each step there will be a different set of features. I understand , feature selection, model building and CV. But how to connect them and in what sequence I'm still not clear. Thank you.


1 Answer 1


Do feature selection inside the cross-validation loop. By this I mean that in each round of the cross-validation procedure, the features to be used should be selected using only the training data.

This said, there are ways to do feature selection that will likely perform better than thresholding correlations, such as the lasso. Consult your favorite machine-learning textbook for a review of feature-selection methods.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.