15
$\begingroup$

I have some questions about the AIC and hope you can help me. I applied model selection (backward, or forward) based on the AIC on my data. And some of the selected variables ended up with a p-values > 0.05. I know that people are saying we should select models based on the AIC instead of the p-value, so seems that the AIC and the p-value are two difference concepts. Could someone tell me what the difference is? What I understand so far is that:

  1. For backward selection using the AIC, suppose we have 3 variables (var1, var2, var3) and the AIC of this model is AIC*. If excluding any one of these three variables would not end up with a AIC which is significantly lower than AIC* (in terms of ch-square distribution with df=1), then we would say these three variables are the final results.

  2. A significant p-value for a variable (e.g. var1) in a three variable model means that the standardized effect size of that variable is significantly different from 0 (according to Wald, or t-test).

What's the fundamental difference between these two methods? How do I interpret it if there are some variables having non-significant p-values in my best model (obtained via the AIC)?

$\endgroup$
13
$\begingroup$

AIC and its variants are closer to variations on $R^2$ then on p-values of each regressor. More precisely, they are penalized versions of the log-likelihood.

You don't want to test differences of AIC using chi-squared. You could test differences of the log-likelihood using chi-squared (if the models are nested). For AIC, lower is better (in most implementations of it, anyway). No further adjustment needed.

You really want to avoid automated model selection methods, if you possibly can. If you must use one, try LASSO or LAR.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Thank you for the answer. Yes, you are right. AIC does not apply any test, instead, it gives a simple measure of how good the model fits the sample and whether the model can be kept simple as well, by adding the -2*loglikelihood with 2*number_of_parameters. Maybe this explains why variables with non-significant p-values were kept in the selected model? $\endgroup$ – tiantianchen Aug 30 '12 at 11:24
  • $\begingroup$ Which model should we choose if we have two models with nearly identical AIC, but in one we have more significant terms than in the other? $\endgroup$ – Agus Camacho Mar 9 '17 at 13:44
  • $\begingroup$ Whichever you want. $\endgroup$ – Peter Flom Mar 9 '17 at 13:57
11
$\begingroup$

In fact using AIC for single-variable-at-a-time stepwise selection is (at least asymptotically) equivalent to stepwise selection using a cut-off for p-values of about 15.7%. (This is quite simple to show - the AIC for the larger model will be smaller if it reduces the log-likelihood by more than the penalty for the extra parameter of 2; this corresponds to choosing the larger model if the p-value in a Wald chi-square is smaller than the tail area of a $\chi^2_1$ beyond 2 ... which is 15.7%)

So it's hardly surprising if you compare it with using some smaller cutoff for p-values that sometimes it includes variables with higher p-values than that cutoff.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ can you point me to a url or reference for the connection between AIC and p-values via Wal chi-square ? Thanks. $\endgroup$ – meh Apr 23 '18 at 16:08
  • $\begingroup$ This is relatively easy to show by using the value of 2 as the critical value, which corresponds to a p-value threshold of 15.73% (when the degrees of freedom of the test is 1, as is the case in stepwise selection using linear regression models and continuous variables). This can be computed as 1-chi2cdf(2,1). $\endgroup$ – George Apr 23 '18 at 16:24
  • $\begingroup$ @aginensky Haven't seen an actual reference, though the connection is straightforward. I imagine I can google one up, hang on. $\endgroup$ – Glen_b Apr 24 '18 at 1:04
  • $\begingroup$ @aginensky Lindsey, J. K. & Jones, B. (1998) Choosing among generalized linear models applied to medical data. Statistics in Medicine, 17 , 59-68. ... see middle of page 62. There would be more. $\endgroup$ – Glen_b Apr 24 '18 at 1:17
  • $\begingroup$ @Glen_b- thanks, i had never seen anything like that before. $\endgroup$ – meh Apr 24 '18 at 4:01
9
$\begingroup$

Note that neither p-values or AIC were designed for stepwise model selection, in fact the assumptions underlying both (but different assumptions) are violated after the first step in a stepwise regression. As @PeterFlom mentioned, LASSO and/or LAR are better alternatives if you feel the need for automated model selection. Those methods pull the estimates that are large by chance (which stepwise rewards for chance) back towards 0 and so tends to be less biased than stepwise (and the remaining bias tends to be more conservative).

A big issue with AIC that is often overlooked is the size of the difference in AIC values, it is all to common to see "lower is better" and stop there (and automated proceedures just emphasise this). If you are comparing 2 models and they have very different AIC values, then there is a clear preference for the model with the lower AIC, but often we will have 2 (or more) models with AIC values that are close to each other, in this case using only the model with the lowest AIC value will miss out on valuable information (and infering things about terms that are in or not in this model but differ in the other similar models will be meaningless or worse). Information from outside the data itself (such as how hard/expensive) it is to collect the set of predictor variables) may make a model with slightly higher AIC more desirable to use without much loss in quality. Another approach is to use a weighted average of the similar models (this will probably result in similar final predictions to the penalized methods like ridge regression or lasso, but the thought process leading to the model might aid in understanding).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you @GregSnow for your answer. May I ask what are the (different) assumptions for p-value and AIC based model selection? Will applying a bi-direction (forward/backward) or trying a full subset more or less solve the problem of finding the local optimal model of simplying using a forward or backward stepwise selection? (although the problem of overfitting always exists in AIC/p-value method and LASSO and/or LAR is a better option) $\endgroup$ – tiantianchen Aug 31 '12 at 9:17
  • $\begingroup$ Since neither p-values or AIC were designed for model selection, they dont have assumptions for model selection. Both were designed to do a single comparison, think about how many comparisons take place in a stepwise regression, do you really think that the "best" step is taken each time? $\endgroup$ – Greg Snow Aug 31 '12 at 17:45
  • $\begingroup$ @GregSnow . My reference for learning AIC was this - stat.cmu.edu/~larry/=stat705/Lecture16.pdf which seems to put AIC in the model selection business. In addition, when I've seen AIC used in time series arima models it was always used for model selection. $\endgroup$ – meh Apr 23 '18 at 16:12
  • $\begingroup$ @aginensky, Yes, AIC (and others) are used for model selection. That does not meen that AIC was designef for model selection, or that it is even appropriate for model selection, or that automated model selection answers a meaningful question. I have used a screwdriver as a hammer before, that does not mean that it is a good idea in general. $\endgroup$ – Greg Snow Apr 23 '18 at 17:20
  • 1
    $\begingroup$ "This paper describes how the problem of statistical model selection can systematically be handled by using an information criteria (AIC) introduced by the author in 1971" from Akaike, " A new look at the statistical model identification" . So even if AIC is a hammer used on a problem that is best solved by a screwdriver, it was the view of the designer of this hammer, that a hammer was the correct way to solve this problem. Correctly or incorrectly, AIC was designed for model selection. I'd be delighted to see a different view of AIC. Feel free to answer this, but I'm done with. $\endgroup$ – meh Apr 23 '18 at 17:38
1
$\begingroup$

My experience with the AIC is that if variables appear non-significant, but still appear in the model with the smallest AIC, those turn out to be possible confounders.

I suggest you check for confounding. Removing such non-significant variables hshould change the magnetude of some remaining estimaated coefficients by more than 25%.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Please explain how OP "can check for confounding." $\endgroup$ – Jim Apr 23 '18 at 13:06
0
$\begingroup$

I think the best model selection is by using MuMIn package. This will be onestep result and you don't have to look for the lowest AIC values. Example:

d<-read.csv("datasource")
library(MuMIn)
fit<-glm(y~x1+x2+x3+x4,family=poisson,data=d)
get.models(dredge(fit,rank="AIC"))[1]
| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Saying what code you might use isn't really answering the question unless you can explain how that addresses the question statistically. In any case nothing in the question is specific to particular software. $\endgroup$ – Nick Cox Dec 8 '14 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.