3
$\begingroup$

I am running a glm model in R, with a dataset with a large number of variables (around 25). I have checked for collinearity, and there is quite some between some of groups of variables, so I have run the tests both with all and with a selected number of variables.

I want to have an explanatory model for an outcome, possibly might progress to have a predictive model later on.

I also have a significant portion of NA's, so although I have 200 rows, I only get 90 observations.

Below is the model with least AIC, chosen with stepAIC. I have chosen a poisson GLM, since the outcome is bound from 0 to 96. Gamma GLM was not possible as I have a number of 0's.

MODEL INFO:
Observations: 90
Dependent Variable: postWOMACpain_6
Type: Linear regression 

MODEL FIT:
χ²(8) = 166.64, p = 0.00
Pseudo-R² (Cragg-Uhler) = 0.29
Pseudo-R² (McFadden) = 0.07
AIC = 411.52, BIC = 436.51 

Standard errors: MLE
---------------------------------------------------
                        Est.   S.E.   t val.      p
-------------------- ------- ------ -------- ------
(Intercept)            -1.66   1.07    -1.55   0.12
anaesthesia.FG          0.94   0.53     1.78   0.08
preWOMACpain            0.26   0.07     3.57   0.00
rs6746030TRUE          -0.83   0.56    -1.48   0.14
rs11898284TRUE          1.33   0.57     2.34   0.02
rs533586TRUE           -2.44   0.70    -3.48   0.00
rs2075572TRUE           1.04   0.64     1.63   0.11
rs609148TRUE            1.77   0.68     2.61   0.01
rs6985606TRUE           0.81   0.55     1.48   0.14
---------------------------------------------------

Estimated dispersion parameter = 5.04 

My query would be:

If the AIC increases when I remove every variable with a p-value > 0.05 in a stepwise manner, then I will end up with only a couple of factors.

For instance, removing anaesthesia.FG, rs6746030, rs6985606 yields:

MODEL INFO:
Observations: 90
Dependent Variable: postWOMACpain_6
Type: Linear regression 

MODEL FIT:
χ²(5) = 138.56, p = 0.00
Pseudo-R² (Cragg-Uhler) = 0.24
Pseudo-R² (McFadden) = 0.06
AIC = 411.50, BIC = 429.00 

Standard errors: MLE
---------------------------------------------------
                        Est.   S.E.   t val.      p
-------------------- ------- ------ -------- ------
(Intercept)            -0.99   0.96    -1.03   0.31
preWOMACpain            0.26   0.07     3.52   0.00
rs11898284TRUE          1.54   0.55     2.79   0.01
rs533586TRUE           -2.04   0.69    -2.97   0.00
rs2075572TRUE           1.10   0.65     1.71   0.09
rs609148TRUE            1.21   0.64     1.90   0.06
---------------------------------------------------

Estimated dispersion parameter = 5.2 

down to:

MODEL INFO:
Observations: 90
Dependent Variable: postWOMACpain_6
Type: Linear regression 

MODEL FIT:
χ²(2) = 90.72, p = 0.00
Pseudo-R² (Cragg-Uhler) = 0.16
Pseudo-R² (McFadden) = 0.04
AIC = 414.86, BIC = 424.86 

Standard errors: MLE
---------------------------------------------------
                        Est.   S.E.   t val.      p
-------------------- ------- ------ -------- ------
(Intercept)            -0.32   0.73    -0.44   0.66
preWOMACpain            0.19   0.07     2.81   0.01
rs11898284TRUE          1.62   0.57     2.84   0.01
---------------------------------------------------

Estimated dispersion parameter = 5.57 

which has a higher AIC, but all variables are all significant.

Your help is much appreciated. All courses I have done do not seem to address this issue, so am not sure how to tackle it properly.

Thanks

$\endgroup$
3
  • 2
    $\begingroup$ Stepwise regression is fraught with problems. $\endgroup$
    – Dave
    Aug 21 at 15:52
  • $\begingroup$ I know. I believe one of my issues is missing data. hence, every time I remove one variable to check the AIC, there would be different number of observations. However, what about my issue with non-significant p-values in the best AIC model? $\endgroup$
    – ssciberras
    Aug 21 at 16:02
  • $\begingroup$ If Stepwise regression is not adequate, what other approach should I use? My research involves checking if a number of variables are associated with an outcome: there might be interactions between the variables, hence the use of a glm $\endgroup$
    – ssciberras
    Aug 21 at 20:13
3
$\begingroup$

I'm not sure what the actual question is here. There seems to be some concern that p-values are "not significant" for the model with the lowest AIC. That is certainly possible, especially when you have missing data.

However, please try not to be too concerned with statistical significance. Since you are interested primarily in inference:

I want to have an explanatory model for an outcome, possibly might progress to have a predictive model later on.

...then it is crucially important that you select the variables for your model in a principled way. Any stepwise procedure is a bad approach to this. More generally any approach based solely on p-values is bad. The model has absolutely no idea which of your variables is the main exposure, competing exposures, potential confounders, colliders or mediators, and as such the model cannot tell you anything about the way these variables are related causally. Moreover, when you adjust for mediators, and colliders, and when you over-adjust for confounders, you can introduce severe bias in the estimates that are of primary concern. See the accepted answer here for further details on principled variable selection for inference:
How do DAGs help to reduce bias in causal inference?

$\endgroup$
4
  • $\begingroup$ Thanks. My main query is if one would remove variables with p > 0.05, even if this increases the AIC? $\endgroup$
    – ssciberras
    Aug 21 at 17:44
  • $\begingroup$ You're welcome. No. Never remove variables based on some arbitrary p-value threshold. $\endgroup$ Aug 21 at 17:47
  • $\begingroup$ Even if p is close to 1? Say 0.92? That would mean that all variables in the best AIC model would be used. I have read your link: it is what i was thinking but need to structure my approach $\endgroup$
    – ssciberras
    Aug 21 at 18:20
  • $\begingroup$ Yes, even if it is close to 1. Under the null hypothesis, p-values are uniformly distributed. $\endgroup$ Aug 21 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.