# What are modern, easily used alternatives to stepwise regression?

I have a dataset with around 30 independent variables and would like to construct a generalized linear model (GLM) to explore the relationship between them and the dependent variable.

I am aware that the method I was taught for this situation, stepwise regression, is now considered a statistical sin.

What modern methods of model selection should be used in this situation?

• Other people mention statistical procedures which may be helpful, but I would first ask whether you have any theory about the strength and shape of the relationship between variables. How big is your sample? Do you have reasons to avoid complex models? Aug 2, 2011 at 19:56
• Has anyone considered model averaging as an alternative to fight the pre-testing bias problem and miss-specification issues? Roughly speaking all variables are potential predictors, and you may estimate the probability for them to be useful. Thus the combined estimator not only improve the forecasting performance, but also produce good properties estimates for the parameters of variables under "scope". Nov 17, 2011 at 12:58
• Shrinkage. Nobody uses stepwise anymore, hopefully Aug 22, 2016 at 17:04

There are several alternatives to Stepwise Regression. The most used I have seen are:

• Expert opinion to decide which variables to include in the model.
• Partial Least Squares Regression. You essentially get latent variables and do a regression with them. You could also do PCA yourself and then use the principal variables.
• Least Absolute Shrinkage and Selection Operator (LASSO).

Both PLS Regression and LASSO are implemented in R packages like

If you only want to explore the relationship between your dependent variable and the independent variables (e.g. you do not need statistical significance tests), I would also recommend Machine Learning methods like Random Forests or Classification/Regression Trees. Random Forests can also approximate complex non-linear relationships between your dependent and independent variables, which might not have been revealed by linear techniques (like Linear Regression).

A good starting point to Machine Learning might be the Machine Learning task view on CRAN:

Machine Learning Task View: http://cran.r-project.org/web/views/MachineLearning.html

• The glmnet package is a very fast implementation of the lasso as well Aug 1, 2011 at 6:31
• I would warn that within the latent variable community, PLSers form a very isolated clique of their own, and were never able to penetrate the serious literature (by which I mean, for instance, asymptotic theory of the least squares estimators in the works of Michael Browne, Peter Bentler, Albert Satorra and Alex Shapiro, and instrumental variable modeling of Ken Bollen, to name the few most important ones). Strangely though, PLS seems to be an acceptable methods in statistics circles, which generally upheld higher standard of rigor than the latent variable modeling community does. Aug 11, 2011 at 7:42
• The Elements of Statistical Learning have a comparison of differend variable selection and shrinkage methods: (OLS,) best subset, ridge, lasso, PLS, PCR. Apr 23, 2012 at 13:53

Another option you might consider for variable selection and regularization is the elastic net. It's implemented in R via the glmnet package.

Model averaging is one way to go (an information-theoretic approach). The R package glmulti can perform linear models for every combination of predictor variables, and perform model averaging for these results.

Don't forget to investigate collinearity between predictor variables first though. Variance Inflation Factors (available in R package "car") are useful here.

• Thanks. Does it really fit all possible models? Even without interactions that's about a billion models in this case. May 1, 2012 at 19:56
• AFAIK it can, but there is a genetic algorithm option that considerably decreases the time it takes to evaluate all models. See www.jstatsoft.org/v34/i12/paper
– OliP
May 2, 2012 at 12:59
• also MuMIn, AICcmodavg packages, although glmulti is cleverer about big model sets. Aug 11, 2012 at 18:54

Interesting discussion. To label stepwise regression as statistical sin is a bit of a religious statement - as long as one knows what they are doing and that the objectives of the exercise is clear, it is definitely a fine approach with its own set of assumptions and, is certainly biased, and does not guarantee optimality, etc. Yet, the same can be said of of lot of other things we do. I have not seen CCA mentioned, which addresses the more fundamental problem of the correlation structure in covariate space, does guarantee optimality, has been around for quite a bit, and it has somewhat of a learning curve. It is implemented on a variety of platforms including R.

@johannes gave an excellent answer. If you are a SAS user, then LASSO is available through PROC GLMSELECT and partial least squares through PROC PLS.

David Cassell and I made a presentation about LASSO (and Least Angle Regression) at a couple of SAS user groups. It's available here