I've read up on the dangers of model selection and came across a neat example on this site for multiple regression whereby forward and backward stepwise selection can result in different final models:


N <- 200000
y <- rnorm(N)
x1 <- y + rnorm(N)
x2 <- y + rnorm(N)
x3 <- y + rnorm(N)
x4 <- rnorm(N)
x5 <- rnorm(N)
x6 <- x1 + x2 + x3 + rnorm(N)
data <- data.frame(y, x1, x2, x3, x4, x5, x6)

fit1 <- lm(y ~ ., data)
fit2 <- lm(y ~ 1, data)

I've since read that ridge regression or lasso regression are alternatives to this form of model selection. But in a simple case like this how would you go about constructing an appropriate model using something like these latter two approaches?


What is frustrating about stepwise regression (as you note) is that the solution is unstable and dependent on the order of entry. Using regularization is often a better approach, and both ridge and lasso are built for this. Alternatively you can use an elastic net, which combines aspects of both. You still won't necessarily obtain one and only one viable solution because you have some design choices on the strength of the regularization parameter (and the mixing parameter in elastic net), but the solutions do tend to be a bit more stable and interpretable.

Look at the glmnet package in R, and consider also the function cv.glmnet to add some rigor, if your dataset is large enough to support it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.