90
$\begingroup$

I have a dataset with around 30 independent variables and would like to construct a generalized linear model (GLM) to explore the relationship between them and the dependent variable.

I am aware that the method I was taught for this situation, stepwise regression, is now considered a statistical sin.

What modern methods of model selection should be used in this situation?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ Other people mention statistical procedures which may be helpful, but I would first ask whether you have any theory about the strength and shape of the relationship between variables. How big is your sample? Do you have reasons to avoid complex models? $\endgroup$
    – Michael Bishop
    Aug 2, 2011 at 19:56
  • 2
    $\begingroup$ Has anyone considered model averaging as an alternative to fight the pre-testing bias problem and miss-specification issues? Roughly speaking all variables are potential predictors, and you may estimate the probability for them to be useful. Thus the combined estimator not only improve the forecasting performance, but also produce good properties estimates for the parameters of variables under "scope". $\endgroup$
    – Dmitrij Celov
    Nov 17, 2011 at 12:58
  • 1
    $\begingroup$ Shrinkage. Nobody uses stepwise anymore, hopefully $\endgroup$
    – Aksakal
    Aug 22, 2016 at 17:04

5 Answers 5

Reset to default
61
$\begingroup$

There are several alternatives to Stepwise Regression. The most used I have seen are:

  • Expert opinion to decide which variables to include in the model.
  • Partial Least Squares Regression. You essentially get latent variables and do a regression with them. You could also do PCA yourself and then use the principal variables.
  • Least Absolute Shrinkage and Selection Operator (LASSO).

Both PLS Regression and LASSO are implemented in R packages like

PLS: http://cran.r-project.org/web/packages/pls/ and

LARS: http://cran.r-project.org/web/packages/lars/index.html

If you only want to explore the relationship between your dependent variable and the independent variables (e.g. you do not need statistical significance tests), I would also recommend Machine Learning methods like Random Forests or Classification/Regression Trees. Random Forests can also approximate complex non-linear relationships between your dependent and independent variables, which might not have been revealed by linear techniques (like Linear Regression).

A good starting point to Machine Learning might be the Machine Learning task view on CRAN:

Machine Learning Task View: http://cran.r-project.org/web/views/MachineLearning.html

Improve this answer
$\endgroup$
3
  • 11
    $\begingroup$ The glmnet package is a very fast implementation of the lasso as well $\endgroup$
    – David J. Harris
    Aug 1, 2011 at 6:31
  • 2
    $\begingroup$ I would warn that within the latent variable community, PLSers form a very isolated clique of their own, and were never able to penetrate the serious literature (by which I mean, for instance, asymptotic theory of the least squares estimators in the works of Michael Browne, Peter Bentler, Albert Satorra and Alex Shapiro, and instrumental variable modeling of Ken Bollen, to name the few most important ones). Strangely though, PLS seems to be an acceptable methods in statistics circles, which generally upheld higher standard of rigor than the latent variable modeling community does. $\endgroup$
    – StasK
    Aug 11, 2011 at 7:42
  • 6
    $\begingroup$ The Elements of Statistical Learning have a comparison of differend variable selection and shrinkage methods: (OLS,) best subset, ridge, lasso, PLS, PCR. $\endgroup$
    – cbeleites unhappy with SX
    Apr 23, 2012 at 13:53
21
$\begingroup$

Another option you might consider for variable selection and regularization is the elastic net. It's implemented in R via the glmnet package.

Improve this answer
$\endgroup$
17
$\begingroup$

Model averaging is one way to go (an information-theoretic approach). The R package glmulti can perform linear models for every combination of predictor variables, and perform model averaging for these results.

See http://sites.google.com/site/mcgillbgsa/workshops/glmulti

Don't forget to investigate collinearity between predictor variables first though. Variance Inflation Factors (available in R package "car") are useful here.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. Does it really fit all possible models? Even without interactions that's about a billion models in this case. $\endgroup$
    – Peter Ellis
    May 1, 2012 at 19:56
  • $\begingroup$ AFAIK it can, but there is a genetic algorithm option that considerably decreases the time it takes to evaluate all models. See www.jstatsoft.org/v34/i12/paper $\endgroup$
    – OliP
    May 2, 2012 at 12:59
  • 3
    $\begingroup$ also MuMIn, AICcmodavg packages, although glmulti is cleverer about big model sets. $\endgroup$
    – Ben Bolker
    Aug 11, 2012 at 18:54
10
$\begingroup$

Interesting discussion. To label stepwise regression as statistical sin is a bit of a religious statement - as long as one knows what they are doing and that the objectives of the exercise is clear, it is definitely a fine approach with its own set of assumptions and, is certainly biased, and does not guarantee optimality, etc. Yet, the same can be said of of lot of other things we do. I have not seen CCA mentioned, which addresses the more fundamental problem of the correlation structure in covariate space, does guarantee optimality, has been around for quite a bit, and it has somewhat of a learning curve. It is implemented on a variety of platforms including R.

Improve this answer
$\endgroup$
8
$\begingroup$

@johannes gave an excellent answer. If you are a SAS user, then LASSO is available through PROC GLMSELECT and partial least squares through PROC PLS.

David Cassell and I made a presentation about LASSO (and Least Angle Regression) at a couple of SAS user groups. It's available here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.