I was doing CS109 lab. There I saw this written:-

"By the way, there is a problem with pre-doing feature selection before doing cross-validation. Ideally one should be doing the feature selection separately in each fold. The reasons for this is basically that there is a high probability that a feature correlates strongly with y just by chance, if there are so many features. How to do this properly will become clear in the homework."

Can, someone explain me, what will happen if features are correlated to y?


What the paragraph tries to say is that if you do feature selection before (rather than inside) the cross-validation loop, the selected features may perform poorly at test time.

The reason for this is that to pre-select features, one typically uses the whole train-val set, but in the subsequent cross-validation step, the same data is used to evaluate the features. The cross-validation scores will therefore overestimate true performance. This effect will be more pronounced if there are many features to choose from, since the chance of many features explaining (correlating with) y on the train-val set is higher.

Note that if the features are selected for each cross-validation fold separately, there is no such problem: for each split, one part of the train-val set is used for feature selection & model fitting, and the other part for evaluation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.