I have a question regarding the process of feature selection, model building and k fold cross validation.
I have a forty features and 200 records data sets. I want to select down to 10-12 features and build a model maybe linear regression. Here is my understand of the whole process.
- In the beginning use all the data, calculate the correlation matrix to find large pair wise correlation and drop some feature highly correlated features
- Use all your data, build different models and use certain measures like R square or Root mean square error to evaluate among different models
- Split you data and do a k fold cross validation to verify if there is over-fitting in your model to evaluate your model. Average the error from each cv as the overall error, compare different model with this overall error.
- Select the model
5. Refit the model with all the data.
But I feel there is some flaw, can someone point out for me. In step 2, my model see all the data. Should I use K-fold CV from beginning? But K fold CV is for model evaluation. Each step there will be a different set of features. I understand , feature selection, model building and CV. But how to connect them and in what sequence I'm still not clear. Thank you.