I have the feeling that when I have a large number of predictors, it will be better to use feature selection regression model such as lasso, to fit the model, and better not use the stepwise feature selection method to select important predictors. The stepwise feature selection methods are only useful when I have only a few of predictors and sufficient samples. I am not sure whether my understanding is correct. If I am correct, why stepwise feature selection method do not perform well when there is a large number of explanatory variables


1 Answer 1


Your probability of spurious correlations grows as the number of dimensions grows, since you have many more "sub manifolds" over which the data can covary. Here's a more in-depth treatment: http://www.jmlr.org/papers/volume17/16-068/16-068.pdf

Therefore, simple stepwise regression will "overfit" to apparent correlations whilst regularized approaches like lasso will tend to correct for this if you use via cross validation to set your regularization parameter. However, even lasso and friends start to have a hard time in very high dimensions (in that they would not be guaranteed to converge to the true model as sample size increases).

However, if you keep in mind that all models are wrong, but some are useful, then as long as your "inconsistent" lasso model is working for predictions, then I wouldn't really worry about the theoretical inconsistency for infinite sample sizes and a perfect model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.