After doing some testing and research, I came across a pitfall of performing feature selection and running cross validation on the results of that process for some model X.
I get the general idea of why this is bad practice due to crating a bias estimate but I was wondering if this also applies to more general feature scoping at the beginning of the modeling process.
For example, if I get a data set with 20 columns and 10k records and perform EDA on the whole thing to pick out say 5 predictors that look good on the basis of correlation or whatever, is there a bias issue here if I use cross validation after fitting model X?
So the steps would be:
- Scope/Select Features
- Train/Test Model X (using k cross)
- Test Model using a holdout set if feasible
I'm having a hard time understanding why this would cause a bias issue since the actual model is not doing the feature selection. It's basically like pretending the other 15 features that I filtered out don't exist in the training or test sets since the scoping was done before fitting any models.
Is this a valid approach or would this also result in upward bias? If so, could someone explain how?
I'm debating on whether or not to just start creating a holdout set as step 1 of any modeling process and perform all the EDA on only the non-holdout set to ensure I'm not using any information to make feature decisions. Let me know if this is a common practice anyone uses. I've also read all the posts relating to this topic but was still unclear how this applies to my outline. Any response is appreciated.