# Why applying model selection using AIC gives me non-significant p-values for the variables

I have some questions about the AIC and hope you can help me. I applied model selection (backward, or forward) based on the AIC on my data. And some of the selected variables ended up with a p-values > 0.05. I know that people are saying we should select models based on the AIC instead of the p-value, so seems that the AIC and the p-value are two difference concepts. Could someone tell me what the difference is? What I understand so far is that:

1. For backward selection using the AIC, suppose we have 3 variables (var1, var2, var3) and the AIC of this model is AIC*. If excluding any one of these three variables would not end up with a AIC which is significantly lower than AIC* (in terms of ch-square distribution with df=1), then we would say these three variables are the final results.

2. A significant p-value for a variable (e.g. var1) in a three variable model means that the standardized effect size of that variable is significantly different from 0 (according to Wald, or t-test).

What's the fundamental difference between these two methods? How do I interpret it if there are some variables having non-significant p-values in my best model (obtained via the AIC)?

-

AIC and its variants are closer to variations on $R^2$ then on p-values of each regressor. More precisely, they are penalized versions of the log-likelihood.

You don't want to test differences of AIC using chi-squared. You could test differences of the log-likelihood using chi-squared (if the models are nested). For AIC, lower is better (in most implementations of it, anyway). No further adjustment needed.

You really want to avoid automated model selection methods, if you possibly can. If you must use one, try LASSO or LAR.

-
Thank you for the answer. Yes, you are right. AIC does not apply any test, instead, it gives a simple measure of how good the model fits the sample and whether the model can be kept simple as well, by adding the -2*loglikelihood with 2*number_of_parameters. Maybe this explains why variables with non-significant p-values were kept in the selected model? –  tiantianchen Aug 30 '12 at 11:24

Note that neither p-values or AIC were designed for stepwise model selection, in fact the assumptions underlying both (but different assumptions) are violated after the first step in a stepwise regression. As @PeterFlom mentioned, LASSO and/or LAR are better alternatives if you feel the need for automated model selection. Those methods pull the estimates that are large by chance (which stepwise rewards for chance) back towards 0 and so tends to be less biased than stepwise (and the remaining bias tends to be more conservative).

A big issue with AIC that is often overlooked is the size of the difference in AIC values, it is all to common to see "lower is better" and stop there (and automated proceedures just emphasise this). If you are comparing 2 models and they have very different AIC values, then there is a clear preference for the model with the lower AIC, but often we will have 2 (or more) models with AIC values that are close to each other, in this case using only the model with the lowest AIC value will miss out on valuable information (and infering things about terms that are in or not in this model but differ in the other similar models will be meaningless or worse). Information from outside the data itself (such as how hard/expensive) it is to collect the set of predictor variables) may make a model with slightly higher AIC more desirable to use without much loss in quality. Another approach is to use a weighted average of the similar models (this will probably result in similar final predictions to the penalized methods like ridge regression or lasso, but the thought process leading to the model might aid in understanding).

-
Thank you @GregSnow for your answer. May I ask what are the (different) assumptions for p-value and AIC based model selection? Will applying a bi-direction (forward/backward) or trying a full subset more or less solve the problem of finding the local optimal model of simplying using a forward or backward stepwise selection? (although the problem of overfitting always exists in AIC/p-value method and LASSO and/or LAR is a better option) –  tiantianchen Aug 31 '12 at 9:17
Since neither p-values or AIC were designed for model selection, they dont have assumptions for model selection. Both were designed to do a single comparison, think about how many comparisons take place in a stepwise regression, do you really think that the "best" step is taken each time? –  Greg Snow Aug 31 '12 at 17:45

In fact using AIC for single-variable-at-a-time stepwise selection is (at least asymptotically) equivalent to stepwise selection using a cut-off for p-values of about 15.7%. (This is quite simple to show - the AIC for the larger model will be smaller if it reduces the log-likelihood by more than the penalty for the extra parameter of 2; this corresponds to choosing the larger model if the p-value in a Wald chi-square is smaller than the tail area of a $\chi^2_1$ beyond 2 ... which is 15.7%)

So it's hardly surprising if you compare it with using some smaller cutoff for p-values that sometimes it includes variables with higher p-values than that cutoff.

-

I think the best model selection is by using MuMIn package. This will be onestep result and you don't have to look for the lowest AIC values. Example:

d<-read.csv("datasource")
library(MuMIn)
fit<-glm(y~x1+x2+x3+x4,family=poisson,data=d)
get.models(dredge(fit,rank="AIC"))[1]

-
Saying what code you might use isn't really answering the question unless you can explain how that addresses the question statistically. In any case nothing in the question is specific to particular software. –  Nick Cox Dec 8 at 11:41