Selecting model in GLM logistic with AICc, p-values for best AICc are all insignificant?

Question

I have one potentially causal predictor and a number of covariates that I tested via AICc model selection in logistic GLM. I found that alone, the causal predictor has a low AICc (~19) and a significant p-value (<0.05). Adding in 1, 2 or 3 of the other covariates gives me a slightly lower AICc (~17) but nonsignificant p-values for the predictors (causal and covariates predictors >>0.05).

Should I be selecting my models based solely on the best AICc? Should I be keeping the p-values in mind (e.g., eliminating a model in either the causal predictor or covariates are insignificant)? Is this a case of overfitting?

Thanks!

Answer 1

You need to start by deciding on what the question is that you are trying to answer and what the science is behind how the data was collected.

The fact that you causal predictor is significant by itself, but not with the other covariates answers some questions, just figure out which are the most relevant. If the data comes from an experiment where the causal variable was properly randomized then the fact that it is significant on its own could be enough. On the other hand if this data came from an observational study and there is reasonable science that one or more of the covariates could cause both the response and the causal variable, then the model with all the terms in it not having the causal variable significant tells you that you need to do more research (preferably a randomized experiment controlling for the covariates). In neither case do you need to do model selection, in both cases you need to understand the science.