I have a conceptual problem understanding how to cross validate stepwise logistic regression. Every time the training set is divided it is very likely that different features are chosen based on the the penter and premove criteria. Should I cross validate using different chosen model every time or Should I find a ground truth and proceed with cross validating over that? I think the latter sounds more reasonable, but I fear that somewhere I'm compromising the test blindness. Help is appreciated.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Stepwise regression often isn't a useful approach, as the results can depend heavily on the particular sample at hand. Why do you need to limit the number of variables, and for what will you use the results of your regression analysis? $\endgroup$– EdMJul 23, 2015 at 17:01
-
$\begingroup$ The aim is high and low classification. The reason I'm using logistic is that some of the features are highly correlated. I don't want to regularize too strongly and I don't want to have to deal with singularities either. Logistic helps in that respect. $\endgroup$– TheodenJul 23, 2015 at 17:34
-
4$\begingroup$ I think your goals would be well-served by using a regularized model, such as elastic net regression, and cross-validate to select the amount of shrinkage with best out-of-sample performance. It achieves variable selection and correction for correlation without any of the drawbacks of stepwise regression. $\endgroup$– Sycorax ♦Jul 23, 2015 at 18:28
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
The Elements of Statistical Learning puts the answer quite clearly (second edition, p. 246):
In general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of modeling steps. In particular, samples must be “left out” before any selection or filtering steps are applied. There is one qualification: initial unsupervised screening steps can be done before samples are left out.
In this type of analysis the problem is that the "ground truth" deduced from your sample might not represent the "ground truth" in the population. Cross-validation can help with generalizing results to the population, but only if all steps of the modeling procedure are repeated for each fold of validation.
As both I and @user777 recommend, you will probably be better off if you use a method other than stepwise selection to deal with your correlated predictor variables. With highly correlated predictors, stepwise selection will almost certainly lead to highly varying choices of predictors from fold to fold. Regularization methods deal with correlated predictors much better. Ridge regression, for example, is essentially a principal-components regression with weights on the components, so that highly correlated variables tend to show up together in the same components.
-
$\begingroup$ Well. Thank you both guys. I will read the section amd i will see how to consider your recommendation. $\endgroup$– TheodenJul 23, 2015 at 19:41
-
$\begingroup$ I'd recommend an ensemble variable selection method called Bootstrap aggregating or Bagging to select your variables to avoid some of the issues others have pointed out. en.wikipedia.org/wiki/Bootstrap_aggregating. $\endgroup$ Jul 24, 2015 at 0:21
$\begingroup$
$\endgroup$
11
The 1970s called. It wants its antiquated, dilapidated stepwise regression back.
The 1990s called. It wants you to employ the ad hoc heuristic methods, including LASSO!!!!, advocated in The Elements of Statistical Learning, as quoted in EdM's answer.
The new millennium called. It's telling you to forget all that ad hoc nonsense and employ a systematic mixed integer optimization approach to choosing the best subsets. This is the way to go, baby. "Best Subset Selection via a Modern Optimization Lens", Bertsimas, King, Mazumder. It will blow The Elements of Statistical Learning recommendations out of the water. Of course, there may not be canned R packages ready to go just yet.
Final version of article later published in The Annals of Statistics (Open Access).
answered Jul 24, 2015 at 0:01
-
2$\begingroup$ The Elements of Statistical Learning presents best-subset regression as the first example of a variable-selection technique (p. 57). In this particular case I fear that any variable-selection technique might lead to problems, due to the correlations among the set of predictor variables. In cross-validation or bootstrapping, best-subset selection seems also likely to lead to a wide range of best subsets, the initial problem noted by the OP. I thus deliberately chose ridge regression rather than LASSO as the example in my answer. $\endgroup$– EdMJul 24, 2015 at 13:57
-
1$\begingroup$ My point is that all the recommendations on the matter in The Elements of Statistical Learning are ad hoc. It's time to move on and use better approaches which modern optimization facilitates. Did you read the paper I linked? Ridge regression was nice in its day. It's time to move on. $\endgroup$ Jul 24, 2015 at 20:27
-
5$\begingroup$ The paper makes a compelling case that mixed-integer optimization is a superior method if you want to find the best k-feature fit (out of p features total) to a response. What isn't so clear, at least from that paper, is whether sparsity of the selected predictor set is always the best goal, particularly for predictive models built from data having multiple correlated predictors in the $n > p$ case (which I took the OP to be). This approach may work better than LASSO for variable selection, but is there documentation that it works better than ridge regression for prediction? $\endgroup$– EdMJul 24, 2015 at 21:06
-
1$\begingroup$ That's left as an exercise for the reader :) $\endgroup$ Jul 24, 2015 at 22:06
-
1$\begingroup$ Just read the paper. Originally I chose forward stepwise selection to train the model since the data and more variables are acquired in time increments and it seemed intuitive to me to refine the model that way. After all the data is acquired n >> p. After reading the paper, I can see how using MIO and callbacks is a great solution for training on the fly. $\endgroup$ Sep 3, 2018 at 15:11