13
$\begingroup$

Stepwise regression had been overused in many biomedical papers in the past but this appears to be improving with better education of its many issues. Many older reviewers however do still ask for it. What are the circumstances where stepwise regression has a role and should be used, if any?

$\endgroup$
  • 6
    $\begingroup$ I often use it when teaching classes to illustrate the idea that it shouldn't be used. $\endgroup$ – gung - Reinstate Monica Jan 25 '17 at 18:23
  • 1
    $\begingroup$ (+1) Thanks @gung I'm not a statistician and have found it difficult to defend this even though I know it is correct. I find it particularly difficult because 1) this very good and often quoted post doesn't go into much citable detail and.... (ctd) $\endgroup$ – bobmcpop Jan 26 '17 at 20:40
  • 2
    $\begingroup$ (ctd) and 2) critics of stepwise often seem to do so for selecting from a large number of variables or data-mining. In the biomedical world, sample size calculations often take into account the number of expected covariates, so that the full models are never that large to begin with, and each var already have some prior "biological" reason to be included. Do you feel that stepwise should equally not be used in these circumstances? $\endgroup$ – bobmcpop Jan 26 '17 at 20:41
  • 3
    $\begingroup$ I do statistical consulting for biomedical research. I don't use stepwise. I haven't had many people ask (they might assume that I would just use it if it would help their project), but when people do ask I tell them that it's invalid & talk about why. $\endgroup$ – gung - Reinstate Monica Jan 26 '17 at 21:04
11
$\begingroup$

I am not aware of situations, in which stepwise regression would be the preferred approach. It may be okay (particularly in its step-down version starting from the full model) with bootstrapping of the whole stepwise process on extremely large datasets with $n>>p$. Here $n$ is the number of observations in an continuous outcome (or number of records with an event in survival analysis) $p$ is the number of candidate predictors including all considered interactions - i.e. when any even small effects become very clear and it does not matter so much how your do your model building (that would mean that $n$ would be much larger than $p$ than by substantially more than the sometimes quoted factor of 20).

Of course the reason most people are tempted to do something like stepwise regression is,

  1. because it is not computationally intensive (if you do not do the proper bootstrapping, but then your results are pretty unreliable),
  2. because it provides clear cut "is in the model" versus "is not in the model" statements (which are very unreliable in standard stepwise regression; something that proper bootstrapping will usually make clear so that these statements will usually not be so clear) and
  3. because often $n$ is smaller, close to or just a bit larger than $p$.

I.e. a method like stepwise regression would (if it had good operating characteristics) be especially attractive in those situations, when it does not have good operating characteristics.

$\endgroup$
  • 3
    $\begingroup$ (+1) Also stepwise & related methods may be appropriate for predictive models in needle-in-a-haystack situations, when a lot of coefficients are negligible & just a few large relative to the error variance. See Example 3 from Tibshirani (1996), Regression shrinkage & selection via the Lasso, JRSS B, 58, 1 - though even here the non-negative garotte wins. $\endgroup$ – Scortchi - Reinstate Monica Jan 25 '17 at 9:32
  • 3
    $\begingroup$ I cannot quite understand the last paragraph. Perhaps it could be rephrased? Also, what about 3.: I do not see a direct argument, perhaps something is supposed to be easy to infer there? $\endgroup$ – Richard Hardy Jan 25 '17 at 17:54
  • 1
    $\begingroup$ To clarify the last paragraph and (3): People use stepwise because of (3) (i.e. to avoid situations where fitting the full model is difficult or leads to $p\approx n$), but that's exactly when it's going to be a terrible method. They use it, because it is not computationally intensive, but to get anything useful out, you have to do extensive bootstrapping (so that's not really an advantage either). And they use it, because it seems to give clear interpretation, but if done properly that is not so clear and you see how much model uncertainty there is (clear interpretation=an illusion). $\endgroup$ – Björn Jan 26 '17 at 8:13
11
$\begingroup$

Two cases in which I would not object to seeing step-wise regression are

  1. Exploratory data analysis
  2. Predictive models

In both these very important use cases, you are not so concerned about traditional statistical inference, so the fact that p-values, etc., are no longer valid is of little concern.

For example, if a research paper said "In our pilot study, we used step-wise regression to find 3 interesting variables out of 1000. In a follow-up study with new data, we showed these 3 interesting variables were strongly correlated with the outcome of interest", I would have no problem with the use of step-wise regression. Similarly, "We used step-wise regression to build a predictive model. This out-preformed alternative model X in our hold-out data set in regards to MSE" is totally fine with me as well.

To be clear, I am not saying that step-wise regression is the best way to approach these problems. But it is easy and may give you satisfactory solutions.

EDIT:

In the comments, there is a question of whether stepwise AIC can actually be useful for prediction. Here's a simulation that shows it doing much better than linear regression with all the covariates, and nearly as well as elastic nets with the penalty chosen by cross-validation.

I wouldn't take this simulation as the end of the discussion; it's not too hard to come up with a scenario in which step-wise AIC will preform worse. But it's really not an unreasonable scenario, and exactly the type of situation that elastic nets are designed for (high correlation of covariates with very few large effects)!

library(leaps)
library(glmnet)
nRows <- 1000
nCols <- 500

# Seed set For reproducibility. 
# Try changing for investigation of reliability of results
set.seed(1)

# Creating heavily correlated covariates
x_firstHalf  <- matrix(rnorm(nRows * nCols / 2), nrow = nRows)
x_secondHalf <- x_firstHalf + 0.5 * 
                matrix(rnorm(nRows * nCols / 2), nrow = nRows) 
x_mat        <- cbind(x_firstHalf, x_secondHalf) + rnorm(nRows)

# Creating beta's. Most will be of very small magnitude
p_large = 0.01
betas <- rnorm(nCols, sd = 0.01) + 
         rnorm(nCols, sd = 4) * rbinom(nCols, size = 1, prob = p_large)
y     <- x_mat %*% betas + rnorm(nRows, sd = 4)

all_data           <- data.frame(y, x_mat)
colnames(all_data) <- c('y', paste('x', 1:nCols, sep = '_'))

# Holding out 25% of data for validation
holdout_index <- 1:(nRows * .25) 
train_data    <- all_data[-holdout_index, ]
validate_data <- all_data[holdout_index, ]

mean_fit <- lm(y ~ 0, data = train_data)
full_fit <- lm(y ~ ., data = train_data)
step_fit <- step(mean_fit, 
                 scope = list(lower = mean_fit, upper = full_fit), 
                 direction = "forward", steps = 20, trace = 0)

glmnet_cvRes <- cv.glmnet(x = as.matrix(train_data[,-1]), 
                          y = as.numeric(train_data$y)   )

full_pred   <- predict(full_fit, validate_data)
step_pred   <- predict(step_fit, validate_data)
glmnet_pred <- predict(glmnet_cvRes, as.matrix(validate_data[,-1]), s='lambda.min')

sd(full_pred - validate_data$y)    # [1] 6.426117
sd(step_pred - validate_data$y)    # [1] 4.233672
sd(glmnet_pred - validate_data$y)  # [1] 4.127171
# Note that stepwise AIC does considerably better than using all covariates 
# in linear regression, and not that much worse than penalized methods
# with cross validation!!

Side note:

I'm really not a fan of stepwise regression for many, many reasons, so I feel somewhat awkward having taken this stance in defense of it. But I merely think it's important to be precise about exactly what I don't like about it.

$\endgroup$
  • 2
    $\begingroup$ What other scenarios is it commonly used in (in biomed literature) except those two indications? I've only come across its use for predictive models, yet it is not advised eg. $\endgroup$ – bobmcpop Jan 25 '17 at 18:15
  • 5
    $\begingroup$ @bobmcpop: the big problem is using p-values, confidence intervals, after stepwise regression, as is mentioned in the paper you cited. Models used solely for prediction (not just models with predictors) generally don't care about p-values, but rather just how much the out-of-sample error is reduced. $\endgroup$ – Cliff AB Jan 25 '17 at 18:25
  • 1
    $\begingroup$ @Björn: well, as I said at the end, I don't think it's generally the best method for doing so by anymeans. But it's not invalid, and you may end up with reasonable results. As such, it's strength is really how easy it is to use: if you have a model that takes covariates and returns likelihoods, you can do step-wise AIC. You may well be able to do better with something like LASSO...but you may not if it's some fancy new model or you're using Excel. $\endgroup$ – Cliff AB Jan 25 '17 at 18:38
  • 4
    $\begingroup$ (+1) I'd have said glmnet was designed to take this sort of situation, among others, in its stride (which it seems to); whereas predictor selection methods without shrinkage are especially attuned to it. Might be interesting to compare approaches when there are "tapering effects" rather than a few large & many tiny ones. $\endgroup$ – Scortchi - Reinstate Monica Jan 27 '17 at 12:42
  • 2
    $\begingroup$ I edited your code to make it easier to read, & easier to copy-and-paste into a code file or console. I hope you like it. If you don't, roll it back w/ my apologies. $\endgroup$ – gung - Reinstate Monica Sep 26 '17 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.