Significance of varibles after stepwise regression

Question

I did stepwise regression with my multiple regression model and using AIC as a measure of fit with the step function in R. Afterwards some variables that the stepwise regression did not eliminate was not significant (> 0.05 p-value). Does this mean i have to take out those variables with large p-values or what is a normal procedure?

Answer 1

The two comments from years ago get at why stepwise regression might not be so helpful for your task. However, it is legitimate to wonder why variables are "insignificant" if the stepwise procedure removes "insignificant" variables.

A potential culprit is multicollinearity of your features. In fact, with multicollinearity, we can have a sky-high F-stat and overall p-value without any individual features being "significant". I think of it like this: the overall hypothesis test knows that some combination of variables contributes to the outcome, but it cannot pin down which one, since the features are related.

In the simulation below, we see a situation where one of the three features is removed by the stepwise procedure, and then the remaining features, which are highly correlated and are not removed, lack significant p-values.

set.seed(2022)
N <- 28
feature_correlation <- 0.9
X <- MASS::mvrnorm(N, c(0, 0), matrix(
    c(1, feature_correlation, 
      feature_correlation, 1), 
    2, 2))
x1 <- X[, 1]
x2 <- X[, 2]
x3 <- rnorm(N)
y <- x1 + x2 + rnorm(N)
L_aic <- lm(y ~ x1 + x2 + x3)
MASS::stepAIC(L_aic)
L <- lm(y ~ x1 + x2)
summary(L)

################################################################################
#
# Results
#
################################################################################

Start:  AIC=13.41
y ~ x1 + x2 + x3

       Df Sum of Sq    RSS    AIC
- x3    1    2.1948 36.157 13.159
- x1    1    2.3282 36.291 13.262
<none>              33.962 13.405
- x2    1    4.6756 38.638 15.017

Step:  AIC=13.16
y ~ x1 + x2

       Df Sum of Sq    RSS    AIC
<none>              36.157 13.159
- x1    1    2.9573 39.115 13.360
- x2    1    3.7779 39.935 13.941

Call:
lm(formula = y ~ x1 + x2)

Coefficients:
(Intercept)           x1           x2  
     0.2867       0.9693       0.9948  

Call:
lm(formula = y ~ x1 + x2)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.44784 -0.67911  0.06944  0.52372  2.63501 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.2867     0.2293   1.250    0.223
x1            0.9693     0.6778   1.430    0.165
x2            0.9948     0.6155   1.616    0.119

Residual standard error: 1.203 on 25 degrees of freedom
Multiple R-squared:  0.7146,    Adjusted R-squared:  0.6918 
F-statistic:  31.3 on 2 and 25 DF,  p-value: 1.56e-07
```