How to cross validate stepwise logistic regression? I have a conceptual problem understanding how to cross validate stepwise logistic regression. Every time the training set is divided it is very likely that different features are chosen based on the the penter and premove criteria. Should I cross validate using different chosen model every time or Should I find a ground truth and proceed with cross validating over that? I think the latter sounds more reasonable, but I fear that somewhere I'm compromising the test blindness.
Help is appreciated.
 A: The 1970s called. It wants its antiquated, dilapidated stepwise regression back.
The 1990s called. It wants you to employ the ad hoc heuristic methods, including LASSO!!!!, advocated in The Elements of Statistical Learning, as quoted in EdM's answer.
The new millennium called. It's telling you to forget all that ad hoc nonsense and employ a systematic mixed integer optimization approach to choosing the best subsets.  This is the way to go, baby. "Best Subset Selection via a Modern Optimization Lens", Bertsimas, King, Mazumder. It will blow The Elements of Statistical Learning recommendations out of the water.  Of course, there may not be canned R packages ready to go just yet.
Final version of article later published in The Annals of Statistics (Open Access).
A: The Elements of Statistical Learning puts the answer quite clearly (second edition, p. 246):

In general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of modeling steps. In particular, samples must be “left out” before any selection or filtering steps are applied. There is one qualification: initial unsupervised screening steps can be done before samples are left out. 

In this type of analysis the problem is that the "ground truth" deduced from your sample might not represent the "ground truth" in the population. Cross-validation can help with generalizing results to the population, but only if all steps of the modeling procedure are repeated for each fold of validation.
As both I and @user777 recommend, you will probably be better off if you use a method other than stepwise selection to deal with your correlated predictor variables. With highly correlated predictors, stepwise selection will almost certainly lead to highly varying choices of predictors from fold to fold. Regularization methods deal with correlated predictors much better. Ridge regression, for example, is essentially a principal-components regression with weights on the components, so that highly correlated variables tend to show up together in the same components.
