Feature Selection Prior to Cross Validating

Question

After doing some testing and research, I came across a pitfall of performing feature selection and running cross validation on the results of that process for some model X.

I get the general idea of why this is bad practice due to crating a bias estimate but I was wondering if this also applies to more general feature scoping at the beginning of the modeling process.

For example, if I get a data set with 20 columns and 10k records and perform EDA on the whole thing to pick out say 5 predictors that look good on the basis of correlation or whatever, is there a bias issue here if I use cross validation after fitting model X?

So the steps would be:

1. Scope/Select Features
2. Train/Test Model X (using k cross)
3. Test Model using a holdout set if feasible

I'm having a hard time understanding why this would cause a bias issue since the actual model is not doing the feature selection. It's basically like pretending the other 15 features that I filtered out don't exist in the training or test sets since the scoping was done before fitting any models.

Is this a valid approach or would this also result in upward bias? If so, could someone explain how?

I'm debating on whether or not to just start creating a holdout set as step 1 of any modeling process and perform all the EDA on only the non-holdout set to ensure I'm not using any information to make feature decisions. Let me know if this is a common practice anyone uses. I've also read all the posts relating to this topic but was still unclear how this applies to my outline. Any response is appreciated.

Thanks,

Answer 1

As I understand it the key issue seems to depend on whether the labels played any role in scoping/feature selection. If they did then the resulting k-fold models will incorporate data leakage from the whole data-set and therefore be vulnerable to producing biased output.

However if the selection is made on the basis of say specific domain knowledge of the importance of the feature or maybe numerical characteristics of the feature (range, variability, distribution) - regardless of any correlation with the response variable - then the approach you describe should not be subject to that kind of bias