I am trying to know the right way of doing feature selection. I am kind of mixed up with feature selection and cross-validation. I am not even sure if I should perform feature selection inside cross-validation or if I should do feature selection first and run cross-validation for model building with selected features.
What is the right to do feature selection when building a model?
Always perform feature selection inside the cross-validation loop on the training data only. The alternative, selecting features first and then splitting into folds, will lead to biased results. If you do that, you are using both the training and the test data to find features that are associated with your target variable - you are using the test data to find features that will work well in the test data, at least in part.
Your test data should never be used in the model training process, it should only be used to test models learned on the training data only. Selecting features on the whole data first integrates all of your potential test data into the model building process, which entirely defeats the purpose of having a test set in the first place.
It's a bit more work, since you need to run feature selection K times instead of just once (once for each of the K folds), but it is the only unbiased way to do it.
