It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?
In one of the top comment, he mentioned:
I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.
Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?
In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?
What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?
Edit: I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?