2
$\begingroup$

In a linear model (regular OLS regression) I am exploring various models of a continuous dependent variable.

When adding some variables, the Akaike Information Criterion (AIC) improves (sometimes dramatically) even though the variable is not close to significant (e.g. p = .49 or even higher).

One reason I can see this might happen is that the other variables in the model become better fit upon addition of the new variable.

Are there other reasons?

As an example:

Model 1: $F_{10, 363} = 4.31, R^2 = .11, AIC (\text{smaller is better}) = 1497.5, -2LL = 1495.5$

Model 2: $F_{11, 362} = 3.93, R^2 = .11, AIC = 1496.1, -2LL = 1494.1$ and the new variable added has $t = -0.10, F = 0.92$.

(This happened with several models)

$\endgroup$
  • $\begingroup$ Why are you talking about AIC specifically? Do other measures, like R-squared, not improve? $\endgroup$ – ttnphns Jul 13 '13 at 15:10
  • 1
    $\begingroup$ If you dealing with nested models (based on "when adding some variables") this should only happen when it's near the borderline (not when the added variables are "not close to significant"). If you're using the likelihood ratio test and you add 1 predictor, significance should happen if the log likelihood jumps by more than 1.92, whereas AIC should be improved if the loglike jumps by more than 1. If $p=.49$ and you added one predictor, this means the likelihood jumped by about $.476/2=.238$, which means the AIC could not have improved. I'd check to make sure there weren't any errors. $\endgroup$ – Macro Jul 13 '13 at 15:13
  • $\begingroup$ @macro Yes, I just added one variable to the model, got no errors, and saw the new variable not close to sig. yet AIC dropped (got better). $\endgroup$ – Peter Flom - Reinstate Monica Jul 13 '13 at 15:16
  • 1
    $\begingroup$ @ttnphns Well, r-squared would always improve when adding a new variable. $\endgroup$ – Peter Flom - Reinstate Monica Jul 13 '13 at 15:16
  • 1
    $\begingroup$ Oh, ok. I see your new update. I think there is an error in the AIC calculation. As you know $AIC = 2L - 2p$. Based on the AIC and likelihood values listed above, $p=1$ in both models, but you've said the second model contains an additional predictor, in which case I think the second AIC should be $1498.1$, which is bigger than the first one. So, neither the likelihood ratio test ($\chi^2 = 1.4$) or the AIC favor the larger model. Also, I think it is a fact that the squared $t$-statistic should equal the corresponding $F$-statistic and $-.1^2 \neq .92$, so I think that may be a typo. $\endgroup$ – Macro Jul 13 '13 at 15:46
1
$\begingroup$

I found part of the solution from colleagues on SAS-L and at SAS:

In SAS PROC MIXED, if you use the default (REML) AIC is affected only by the random effects and covariance structure. If you use ML, then AIC is affected by the fixed effects (but not the covariance structure).

Apparently, the reasons for doing this (which still seems counterintuitive to me) is found in Littell et al and also in a paper by Vonesh and Chinchilli but I have not read these.

However, I have a new example where the LL improves a lot with the addition of a nonsignificant variable; I will post that as a new question.

$\endgroup$
  • $\begingroup$ I don't understand "I have a new example where the LL improves a lot with the addition of a nonsignificant variable". If the LL improved a lot (i.e. more than 1.92) when added to a model where no other variables have been removed, then the variable is significant (at least for large sample sizes). This is the classical theory related to the likelihood ratio test... $\endgroup$ – Macro Jul 15 '13 at 18:18
  • 1
    $\begingroup$ @macro See my new question. Something else is happening. I gave SAS code and results. $\endgroup$ – Peter Flom - Reinstate Monica Jul 15 '13 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.