16
$\begingroup$

I have tried calculating the AIC of a linear regression in R but without using the AIC function, like this:

lm_mtcars <- lm(mpg ~ drat, mtcars)

nrow(mtcars)*(log((sum(lm_mtcars$residuals^2)/nrow(mtcars))))+(length(lm_mtcars$coefficients)*2)
[1] 97.98786

However, AIC gives a different value:

AIC(lm_mtcars)
[1] 190.7999

Could somebody tell me what I'm doing wrong?

| cite | improve this question | |
$\endgroup$
  • 5
    $\begingroup$ (without checking your answer yet): You're not necessarily doing anything wrong, since likelihood is actually only defined up to a multiplicative constant; two people can calculate log-likelihood and get different numbers (but differences in log-likelihood is the same). $\endgroup$ – Glen_b Feb 20 '14 at 20:26
  • 1
    $\begingroup$ Hong Oois answer is related to this question, I think. The formula that the function AIC uses is -2*as.numeric(logLik(lm_mtcars))+2*(length(lm_mtcars$coefficients)+1). $\endgroup$ – COOLSerdash Feb 20 '14 at 20:27
  • $\begingroup$ luciano: The "+1" in that formula @COOLSerdash points to arises from the variance parameter term. Note also that the function logLik says that for lm models it includes 'all constants' ... so there'll be a log(2*pi) in there somewhere $\endgroup$ – Glen_b Feb 20 '14 at 21:30
  • 1
    $\begingroup$ @Glen_b: Why say likelihood's defined only up to an multiplicative constant? After all, when comparing non-nested models from different families of distribution (e.g. with AIC, or with the Cox test), you need to remember that constant. $\endgroup$ – Scortchi - Reinstate Monica♦ Feb 20 '14 at 21:43
  • $\begingroup$ @Scortchi the definition isn't mine! You'll have to take it up with R.A.Fisher. It's been that way from the start, I think (1921). That it's still defined that way, at least in the continuous case, see here, for example, at the sentence beginning 'More precisely,'. $\endgroup$ – Glen_b Feb 20 '14 at 21:52
19
$\begingroup$

Note that the help on the function logLik in R says that for lm models it includes 'all constants' ... so there will be a log(2*pi) in there somewhere, as well as another constant term for the exponent in the likelihood. Also, you can't forget to count the fact that $\sigma^2$ is a parameter.

$\cal L(\hat\mu,\hat\sigma)=(\frac{1}{\sqrt{2\pi s_n^2}})^n\exp({-\frac{1}{2}\sum_i (e_i^2/s_n^2)})$

$-2\log \cal{L} = n\log(2\pi)+n\log{s_n^2}+\sum_i (e_i^2/s_n^2)$

$= n[\log(2\pi)+\log{s_n^2}+1]$

$\text{AIC} = 2p -2\log \cal{L}$

but note that for a model with 1 independent variable, p=3 (the x-coefficient, the constant and $\sigma^2$)

Which means this is how you get their answer:

nrow(mtcars)*(log(2*pi)+1+log((sum(lm_mtcars$residuals^2)/nrow(mtcars))))
       +((length(lm_mtcars$coefficients)+1)*2)
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Why in your calculation of $s^2$ are you only dividing by $n$ and not $n-p$? $\endgroup$ – Luke Thorburn Sep 28 '17 at 1:00
  • 1
    $\begingroup$ See the definition of AIC: $-2\log\mathcal{L}(\hat\theta)+2p$ where the vector of parameters, $\theta$ are evaluated at the maximum (i.e. all the elements of $\hat\theta$ are MLEs); e.g. see Wikipedia Akaike information criterion: Definition. If you're not dividing by $n$ there in the computation of $\hat{\sigma}^2$, you're not calculating the MLE of $\sigma^2$ and so not really computing AIC -- in effect you'd be adjusting twice for the effect of fitting parameters. (Yes, lots of people do it wrong) $\endgroup$ – Glen_b Sep 28 '17 at 1:01
  • $\begingroup$ Is there a typo in the second equation? Should it be $-2\log \cal{L} = n\log(2\pi)+n\log{s_n}+\sum_i (e_i^2/s_n^2)$ Ok I see, you're using $\sqrt{2\pi s^2_n}$ $\endgroup$ – rhody Sep 26 '19 at 22:08
12
$\begingroup$

The AIC function gives $2k -2 \log L$, where $L$ is the likelihood & $k$ is the number of estimated parameters (including the intercept, & the variance). You're using $n \log \frac{S_{\mathrm{r}}}{n} + 2(k-1)$, where $S_{\mathrm{r}}$ is the residual sum of squares, & $n$ is the sample size. These formulæ differ by an additive constant; so long as you're using the same formula & looking at differences in AIC between different models where the constants cancel, it doesn't matter.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.