Why does model selection using AIC yield non-significant p-values for the variables?

Question

I have some questions about the AIC and hope you can help me. I applied model selection (backward, or forward) based on the AIC on my data. And some of the selected variables ended up with a p-values > 0.05. I know that people are saying we should select models based on the AIC instead of the p-value, so seems that the AIC and the p-value are two difference concepts. Could someone tell me what the difference is? What I understand so far is that:

1. For backward selection using the AIC, suppose we have 3 variables (var1, var2, var3) and the AIC of this model is AIC*. If excluding any one of these three variables would not end up with a AIC which is significantly lower than AIC* (in terms of ch-square distribution with df=1), then we would say these three variables are the final results.

2. A significant p-value for a variable (e.g. var1) in a three variable model means that the standardized effect size of that variable is significantly different from 0 (according to Wald, or t-test).

What's the fundamental difference between these two methods? How do I interpret it if there are some variables having non-significant p-values in my best model (obtained via the AIC)?

Answer 1

AIC and its variants are closer to variations on $R^2$ then on p-values of each regressor. More precisely, they are penalized versions of the log-likelihood.

You don't want to test differences of AIC using chi-squared. You could test differences of the log-likelihood using chi-squared (if the models are nested). For AIC, lower is better (in most implementations of it, anyway). No further adjustment needed.

You really want to avoid automated model selection methods, if you possibly can. If you must use one, try LASSO or LAR.

Answer 2

In fact using AIC for single-variable-at-a-time stepwise selection is (at least asymptotically) equivalent to stepwise selection using a cut-off for p-values of about 15.7%. (This is quite simple to show - the AIC for the larger model will be smaller if it reduces the log-likelihood by more than the penalty for the extra parameter of 2; this corresponds to choosing the larger model if the p-value in a Wald chi-square is smaller than the tail area of a $\chi^2_1$ beyond 2 ... which is 15.7%)

So it's hardly surprising if you compare it with using some smaller cutoff for p-values that sometimes it includes variables with higher p-values than that cutoff.

Answer 3

Note that neither p-values or AIC were designed for stepwise model selection, in fact the assumptions underlying both (but different assumptions) are violated after the first step in a stepwise regression. As @PeterFlom mentioned, LASSO and/or LAR are better alternatives if you feel the need for automated model selection. Those methods pull the estimates that are large by chance (which stepwise rewards for chance) back towards 0 and so tends to be less biased than stepwise (and the remaining bias tends to be more conservative).

A big issue with AIC that is often overlooked is the size of the difference in AIC values, it is all to common to see "lower is better" and stop there (and automated proceedures just emphasise this). If you are comparing 2 models and they have very different AIC values, then there is a clear preference for the model with the lower AIC, but often we will have 2 (or more) models with AIC values that are close to each other, in this case using only the model with the lowest AIC value will miss out on valuable information (and infering things about terms that are in or not in this model but differ in the other similar models will be meaningless or worse). Information from outside the data itself (such as how hard/expensive) it is to collect the set of predictor variables) may make a model with slightly higher AIC more desirable to use without much loss in quality. Another approach is to use a weighted average of the similar models (this will probably result in similar final predictions to the penalized methods like ridge regression or lasso, but the thought process leading to the model might aid in understanding).

Answer 4

My experience with the AIC is that if variables appear non-significant, but still appear in the model with the smallest AIC, those turn out to be possible confounders.

I suggest you check for confounding. Removing such non-significant variables hshould change the magnetude of some remaining estimaated coefficients by more than 25%.

Answer 5

I think the best model selection is by using MuMIn package. This will be onestep result and you don't have to look for the lowest AIC values. Example:

d<-read.csv("datasource")
library(MuMIn)
fit<-glm(y~x1+x2+x3+x4,family=poisson,data=d)
get.models(dredge(fit,rank="AIC"))[1]