Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a dataset with around 30 independent variables and would like to construct a GLM to explore the relationship between them and the dependent variable.

I am aware that the method I was taught for this situation, stepwise regression, is now considered a statistical sin. What modern methods of model selection should be used in this situation?

share|improve this question
Other people mention statistical procedures which may be helpful, but I would first ask whether you have any theory about the strength and shape of the relationship between variables. How big is your sample? Do you have reasons to avoid complex models? –  Michael Bishop Aug 2 '11 at 19:56
Has anyone considered model averaging as an alternative to fight the pre-testing bias problem and miss-specification issues? Roughly speaking all variables are potential predictors, and you may estimate the probability for them to be useful. Thus the combined estimator not only improve the forecasting performance, but also produce good properties estimates for the parameters of variables under "scope". –  Dmitrij Celov Nov 17 '11 at 12:58

5 Answers 5

up vote 37 down vote accepted

There are several alternatives to Stepwise Regression. The most used I have seen are:

  • Expert opinion to decide which variables to include in the model.
  • Partial Least Squares Regression. You essentially get latent variables and do a regression with them. You could also do PCA yourself and then use the principal variables.
  • Least Absolute Shrinkage and Selection Operator (LASSO).

Both PLS Regression and LASSO are implemented in R packages like

PLS: http://cran.r-project.org/web/packages/pls/ and

LARS: http://cran.r-project.org/web/packages/lars/index.html

If you only want to explore the relationship between your dependent variable and the independent variables (e.g. you do not need statistical significance tests), I would also recommend Machine Learning methods like Random Forests or Classification/Regression Trees. Random Forests can also approximate complex non-linear relationships between your dependent and independent variables, which might not have been revealed by linear techniques (like Linear Regression).

A good starting point to Machine Learning might be the Machine Learning task view on CRAN:

Machine Learning Task View: http://cran.r-project.org/web/views/MachineLearning.html

share|improve this answer
The glmnet package is a very fast implementation of the lasso as well –  David J. Harris Aug 1 '11 at 6:31
I would warn that within the latent variable community, PLSers form a very isolated clique of their own, and were never able to penetrate the serious literature (by which I mean, for instance, asymptotic theory of the least squares estimators in the works of Michael Browne, Peter Bentler, Albert Satorra and Alex Shapiro, and instrumental variable modeling of Ken Bollen, to name the few most important ones). Strangely though, PLS seems to be an acceptable methods in statistics circles, which generally upheld higher standard of rigor than the latent variable modeling community does. –  StasK Aug 11 '11 at 7:42
The Elements of Statistical Learning have a comparison of differend variable selection and shrinkage methods: (OLS,) best subset, ridge, lasso, PLS, PCR. –  cbeleites Apr 23 '12 at 13:53

Another option you might consider for variable selection and regularization is the elastic net. It's implemented in R via the glmnet package.

share|improve this answer

Model averaging is one way to go (an information-theoretic approach). The R package glmulti can perform linear models for every combination of predictor variables, and perform model averaging for these results.

See http://sites.google.com/site/mcgillbgsa/workshops/glmulti

Don't forget to investigate collinearity between predictor variables first though. Variance Inflation Factors (available in R package "car") are useful here.

share|improve this answer
Thanks. Does it really fit all possible models? Even without interactions that's about a billion models in this case. –  Peter Ellis May 1 '12 at 19:56
AFAIK it can, but there is a genetic algorithm option that considerably decreases the time it takes to evaluate all models. See www.jstatsoft.org/v34/i12/paper –  OliP May 2 '12 at 12:59
also MuMIn, AICcmodavg packages, although glmulti is cleverer about big model sets. –  Ben Bolker Aug 11 '12 at 18:54

@johannes gave an excellent answer. If you are a SAS user, then LASSO is available through PROC GLMSELECT and partial least squares through PROC PLS.

David Cassell and I made a presentation about LASSO (and Least Angle Regression) at a couple of SAS user groups. It's available here

share|improve this answer

Interesting discussion. To label stepwise regression as statistical sin is a bit of a religious statement - as long as one knows what they are doing and that the objectives of the exercise is clear, it is definitely a fine approach with its own set of assumptions and, is certainly biased, and does not guarantee optimality, etc. Yet, the same can be said of of lot of other things we do. I have not seen CCA mentioned, which addresses the more fundamental problem of the correlation structure in covariate space, does guarantee optimality, has been around for quite a bit, and it has somewhat of a learning curve. It is implemented on a variety of platforms including R.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.