4
$\begingroup$

I want to run a binary logistic regression to understanding (modeling) factors affecting nest-site selection in a bird species. I have Presence/Absence data and 13 predictors. The sample size is small. 32 Presence points and 64 Absence points.

I want to run a stepwise binary logistic regression in R manually. I have 2 questions about it.

Which method can be better? Forward or Backward?

Which statistic is more important for removing or adding variables at each step? AIC or p-value of ANOVA (Wald test) ?

$\endgroup$
14
  • 8
    $\begingroup$ Why do you want to do a stepwise regression? $\endgroup$
    – Dave
    Feb 22, 2023 at 22:25
  • 6
    $\begingroup$ What do you mean by “significant effect”? To calculate statistical significance under the regular meaning, you don't need either of those. $\endgroup$
    – Tim
    Feb 22, 2023 at 22:32
  • 6
    $\begingroup$ What the other two posters are trying to tell you is that you don’t need to do stepwise regression. That method is demonstrably bad, and fails to reliably do what it purports to do (Identify variables impacting the outcome). The widely given but rarely accepted advice is to avoid it and just run a regression with all variables (assuming your observation per variable ratio is fairly high. Else you have another problem). $\endgroup$ Feb 22, 2023 at 22:41
  • 9
    $\begingroup$ It's almost always more effective to tell us what problem you want to solve rather than asking how to apply a particular procedure to it. We will understand your problem better and you will avoid getting good advice on doing the wrong thing! $\endgroup$
    – whuber
    Feb 22, 2023 at 22:42
  • 5
    $\begingroup$ Re your last comment: unfortunately, statistics are misused EXTREMELY OFTEN, usually not through malice, but because people don't know better, and/or take their cue from earlier misinformed papers. Psychologists recently got a rude wake-up call in their "replicability crisis" and seem to be slowly adopting sounder methods. Machine Learners are perpetuating their own kinds of voodoo statistics. Disciplines differ very much in terms of quality of statistics. $\endgroup$ Feb 23, 2023 at 15:31

2 Answers 2

20
$\begingroup$

Because I'm frankly tired of answering questions about stepwise without something of my own to point to, consider the following.

I'm going to simulate a logistic regression with 10 parameters. The variables $x_1, x_2, x_3$ are all independent and have log odds ratios of 0.1, 0.2, and 0.5.

The variables $x_4, x_5, x_6$ have no effect on the log odds, but are correlated with the variables $x_1, x_2, x_3$ like

$$ \operatorname{Cor}(x_j, x_{j+3}) = 0.3\cdot j $$

So $x_1$ and $x_4$ are correlated, but $x_2$ and $x_4$ are not.

Lastly, the variables $x_7, x_8, x_9$ are independent of all others and have no effect on the log odds.

Those who use stepwise regression seem to think that it can select relevant variables. So, if that were true, then surely stepwise regression could select the right variables for this problem, right?

In 1000 simulations from this process, using 1000 observations:

  • $x_1$ is selected 37% of the time
  • $x_2$ is selected 70% of the time, and
  • $x_3$ is selected 86% of the time

It would appear that larger effects are being selected with larger frequency. But how often is the true model selected? A whopping 9% of the time.

Let me repeat that. Almost 90% of the time, you're selecting the wrong model; you're including a variable which actually has 0 impact on the outcome or excluding a variable which does impact the outcome. This is especially damning because all assumptions of the model are met as best as they can be met. All variables affecting the outcome are eligible to be selected and they are all linear in the log odds. Nothing is wrong with the model, it's the selection that is causing this apparent mal performance.

There are a host of problems with stepwise regression (I'll link them here for you to read). It's very clear that there is more nuance to what stepwise regression is doing. It isn't selecting the right model anywhere near enough to justify its use.

Now, I think people may take objection with what I've argued here. "Demetri, we might not even select the right model even using our scientific judgement. It isn't fair to criticize stepwise regression on those grounds". Ok, maybe, but that is one nail removed from the coffin and some 10 more from that list I've linked.

Anyway, don't use stepwise.

Code

library(tidyverse)

N <- 1000
q <- 9
Sigma <- diag(q)
Sigma[1, 4] <- Sigma[4, 1] <- 0.3
Sigma[2, 5] <- Sigma[5, 2] <- 0.6
Sigma[3, 6] <- Sigma[6, 3] <- 0.9

results <- map_dfr(1:1000, ~{
  # Simulate data
  X <- MASS::mvrnorm(N, mu=rep(0, q), Sigma = Sigma)
  beta <- rep(0, 9)
  beta[1:3] <- c( 0.1, 0.2, 0.5)
  y <- rbinom(N, 1, plogis(X%*% beta - 2))
  d <- as_tibble(X) %>% 
       mutate(y=y)
  
  # Fit a model and do stepwise regression
  full.fit <- glm(y~.,, data=d, family = binomial)
  step.fit <- MASS::stepAIC(full.fit, direction = 'both', trace = F)
  
  # Grab which variables were selected
  selected_var <- names(coef(step.fit))
  # Determine if the right model was selected
  correct_model <- c( "(Intercept)","V1", "V2", "V3")
  correct_model_selected <- identical(selected_var,correct_model)
  
  # Determine which of the variables were selected from the sample
  vrs <- str_c("V", 1:10) 
  outcomes<-vrs %in% selected_var
  
  names(outcomes) <- vrs
  
  as.data.frame(t(outcomes)) %>% 
    mutate(correct_model = correct_model_selected)
})

summarise_all(results, mean)

$\endgroup$
1
  • $\begingroup$ +1 Also: \m..m/ (Too much metal for one hand! :) $\endgroup$
    – Alexis
    Feb 23, 2023 at 18:33
16
$\begingroup$

NEITHER: Do not use stepwise model building

I encourage you to practice searching for answers to similar questions first. And secondly, this kind of automated model-build has been roundly and soundly thrashed. It is, to crib from Monte Python, an ex-method!

There are many similar answers on CrossValidated providing the same insight.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.