19
$\begingroup$

I have tried calculating the AIC of a linear regression in R but without using the AIC function, like this:

lm_mtcars <- lm(mpg ~ drat, mtcars)

nrow(mtcars)*(log((sum(lm_mtcars$residuals^2)/nrow(mtcars))))+(length(lm_mtcars$coefficients)*2)
[1] 97.98786

However, AIC gives a different value:

AIC(lm_mtcars)
[1] 190.7999

Could somebody tell me what I'm doing wrong?

Improve this question
$\endgroup$
8
  • 5
    $\begingroup$ (without checking your answer yet): You're not necessarily doing anything wrong, since likelihood is actually only defined up to a multiplicative constant; two people can calculate log-likelihood and get different numbers (but differences in log-likelihood is the same). $\endgroup$
    – Glen_b
    Feb 20, 2014 at 20:26
  • 1
    $\begingroup$ Hong Oois answer is related to this question, I think. The formula that the function AIC uses is -2*as.numeric(logLik(lm_mtcars))+2*(length(lm_mtcars$coefficients)+1). $\endgroup$
    – COOLSerdash
    Feb 20, 2014 at 20:27
  • $\begingroup$ luciano: The "+1" in that formula @COOLSerdash points to arises from the variance parameter term. Note also that the function logLik says that for lm models it includes 'all constants' ... so there'll be a log(2*pi) in there somewhere $\endgroup$
    – Glen_b
    Feb 20, 2014 at 21:30
  • 1
    $\begingroup$ @Glen_b: Why say likelihood's defined only up to an multiplicative constant? After all, when comparing non-nested models from different families of distribution (e.g. with AIC, or with the Cox test), you need to remember that constant. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Feb 20, 2014 at 21:43
  • 1
    $\begingroup$ @Glen_b: Fair enough. There's a post here on how to rigorously define likelihood. For the moment (i.e. till I miraculously become clever enough to understand it all) at least I like gui11aumes's comment: "Personally I prefer to call it up [equality up to a multiplicative constant] when needed rather than hard code it in the definition." $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Feb 20, 2014 at 22:28

2 Answers 2

Reset to default
21
$\begingroup$

Note that the help on the function logLik in R says that for lm models it includes 'all constants' ... so there will be a log(2*pi) in there somewhere, as well as another constant term for the exponent in the likelihood. Also, you can't forget to count the fact that $\sigma^2$ is a parameter.

$\cal L(\hat\mu,\hat\sigma)=(\frac{1}{\sqrt{2\pi s_n^2}})^n\exp({-\frac{1}{2}\sum_i (e_i^2/s_n^2)})$

$-2\log \cal{L} = n\log(2\pi)+n\log{s_n^2}+\sum_i (e_i^2/s_n^2)$

$= n[\log(2\pi)+\log{s_n^2}+1]$

$\text{AIC} = 2p -2\log \cal{L}$

but note that for a model with 1 independent variable, p=3 (the x-coefficient, the constant and $\sigma^2$)

Which means this is how you get their answer:

nrow(mtcars)*(log(2*pi)+1+log((sum(lm_mtcars$residuals^2)/nrow(mtcars))))
       +((length(lm_mtcars$coefficients)+1)*2)
Improve this answer
$\endgroup$
3
  • $\begingroup$ Why in your calculation of $s^2$ are you only dividing by $n$ and not $n-p$? $\endgroup$
    – Luke Thorburn
    Sep 28, 2017 at 1:00
  • 1
    $\begingroup$ See the definition of AIC: $-2\log\mathcal{L}(\hat\theta)+2p$ where the vector of parameters, $\theta$ are evaluated at the maximum (i.e. all the elements of $\hat\theta$ are MLEs); e.g. see Wikipedia Akaike information criterion: Definition. If you're not dividing by $n$ there in the computation of $\hat{\sigma}^2$, you're not calculating the MLE of $\sigma^2$ and so not really computing AIC -- in effect you'd be adjusting twice for the effect of fitting parameters. (Yes, lots of people do it wrong) $\endgroup$
    – Glen_b
    Sep 28, 2017 at 1:01
  • $\begingroup$ Is there a typo in the second equation? Should it be $-2\log \cal{L} = n\log(2\pi)+n\log{s_n}+\sum_i (e_i^2/s_n^2)$ Ok I see, you're using $\sqrt{2\pi s^2_n}$ $\endgroup$
    – rhody
    Sep 26, 2019 at 22:08
13
$\begingroup$

The AIC function gives $2k -2 \log L$, where $L$ is the likelihood & $k$ is the number of estimated parameters (including the intercept, & the variance). You're using $n \log \frac{S_{\mathrm{r}}}{n} + 2(k-1)$, where $S_{\mathrm{r}}$ is the residual sum of squares, & $n$ is the sample size. These formulæ differ by an additive constant; so long as you're using the same formula & looking at differences in AIC between different models where the constants cancel, it doesn't matter.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.