25
$\begingroup$

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work?

I just came around this question when I did comparison like this:

> uu0 = lm(log(usili) ~ rok)
> uu1 = lm(usili ~ rok)
> AIC(uu0)
[1] 3192.14
> AIC(uu1)
[1] 14277.29

This way I justified the log transformation of variable usili. But I don't know if I can AIC-compare models when for example the dependent variable is different?

Ideal answer would include the list of prerequisites (mathematical assumptions).

$\endgroup$
29
$\begingroup$

You can not compare the two models as they do not model the same variable (as you correctly recognise yourself). Nevertheless AIC should work when comparing both nested and nonnested models.

Just a reminder before we continue: a Gaussian log-likelihood is given by

$$ \log(L(\theta)) =-\frac{|D|}{2}\log(2\pi) -\frac{1}{2} \log(|K|) -\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu), $$

$K$ being the covariance structure of your model, $|D|$ the number of points in your datasets, $\mu$ the mean response and $x$ your dependent variable.

More specifically AIC is calculated to be equal to $2k - 2 \log(L)$, where $k$ is the number of fixed effects in your model and $L$ your likelihood function [1]. It practically compares trade-off between variance ($2k$) and bias ($2\log(L)$) in your modelling assumptions. As such in your case it would compare two different log-likelihood structures when it came to the bias term. That is because when you calculate your log-likelihood practically you look at two terms: a fit term, denoted by $-\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu)$, and a complexity penalization term, denoted by $-\frac{1}{2} \log(|K|)$. Therefore you see that your fit term is completely different between the two models; in the first case you compare the residuals from the raw data and in the other case the residuals of the logged data.

Aside Wikipedia, AIC is also defined to equate: $|D| \log\left(\frac{RSS}{|D|}\right) + 2k$ [3]; this form makes it even more obvious why different models with different dependent variable are not comparable. The RSS is the two case is just incomparable between the two.

Akaike's original paper [4] is actually quite hard to grasp (I think). It is based on KL divergence (difference between two distributions roughly speaking) and works its way on proving how you can approximate the unknown true distribution of your data and compare that to the distribution of the data your model assumes. That's why "smaller AIC score is better"; you are closer to the approximate true distribution of your data.

So to bring it all together the obvious things to remember when using AIC are three [2,5] :

  1. You can not use it to compare models of different data sets.

  2. You should use the same response variables for all the candidate models.

  3. You should have $|D| >> k$, because otherwise you do not get good asymptotic consistency.

Sorry to break the bad news to you but using AIC to show you are choosing one dependent variable over another is not a statistically sound thing to do. Check the distribution of your residuals in both models, if the logged data case has normally distributed residuals and the raw data case doesn't, you have all the justification you might ever need. You might also want to check if your raw data correspond to a lognormal, that might be enough of a justification also.

For strict mathematical assumptions the game is KL divergence and information theory...

Ah, and some references:

  1. http://en.wikipedia.org/wiki/Akaike_information_criterion
  2. Akaike Information Criterion, Shuhua Hu, (Presentation p.17-18)
  3. Applied Multivariate Statistical Analysis, Johnson & Wichern, 6th Ed. (p. 386-387)
  4. A new look at the statistical model identification, H. Akaike, IEEE Transactions on Automatic Control 19 (6): 716–723 (1974)
  5. Model Selection Tutorial #1: Akaike’s Information Criterion, D. Schmidt and E. Makalic, (Presentation p.39)
$\endgroup$
  • $\begingroup$ thanks! I did not understood the math but I got the core of the message. However, can you please list all the prerequisites needed for AIC model comparison? Just to be sure I will not make another mistake next time. I'll go and check them one by one. $\endgroup$ – Curious Jan 28 '13 at 22:16
  • 1
    $\begingroup$ I am afraid I don't have a "check-list" as such. Ref.[2] has a quite comprehensive list if you are interested though. Main things to remember are that: 1. because AIC is an asymptotically efficient model selection criterion you need $|D|$ to be significantly greater than $p$ and 2. you can to use it only to compare models of the same dependent data. Mathematically speaking you want $L(\theta)$ to be twice differentiable, every candidate model $\theta$ to be mapped a unique $p(x|\theta)$ and your ML estimates to be consistent, but I think these assumptions are an overkill to show in a paper... $\endgroup$ – usεr11852 Jan 28 '13 at 22:45
  • 1
    $\begingroup$ thank you for adding list of those 3 assumptions to the answer! That's what I needed. $\endgroup$ – Curious Jan 29 '13 at 9:04
  • 1
    $\begingroup$ Looking at your answer again: your point 1. "You can not use it to compare models of different data sets". What you mean by "data set"? What if I change the set of dependent variables? I guess that in that case AIC should be still comparable? Can you please update your answer to clarify this? $\endgroup$ – Curious Jul 24 '13 at 7:11
  • 1
    $\begingroup$ (Sorry for the very late reply!) I think you want to say independent variables... If you change your dependent variable you are messing up with your $RSS$ once more as the "model fits" (roughly speaking, $\mu$) are not compared against the same $x$. (Take your time answering @Curious, I won't expecting anything before mid July! :D ) $\endgroup$ – usεr11852 Jan 23 '14 at 23:56
10
$\begingroup$

You should be able to compare using AIC in principle, just that the number called "AIC" is not the number you need. You are comparing normal vs log-normal distributions. Now the AIC from model uu0 is basically just missing the "jacobian" of the log transformation. For a log normal model, this is simply $\prod_i y_i^{-1} $. To convert this to AIC you need to take negative twice log of this term, which means that you need to add $2\sum_i\log (y_i)$ to the AIC number for uu0. So you should have AIC (uu0)+2*sum (log (usili)) being compared with AIC (uu1)

$\endgroup$
  • $\begingroup$ I don't understand what you follow with your attempt to "correct" AIC somehow and what did you actually get by it (how to interpret your result). Anyway, don't dig into this, it doesn't matter because my question was about something completely different: what are the general prerequisites for the AIC (actual, uncorrected) to be sensibly comparable. Don't focus on this particular example, it's just an example of the general thing. $\endgroup$ – Curious May 31 '14 at 16:03
  • 1
    $\begingroup$ @curious - my point is that my "corrected AIC" is the actual AIC, and the thing you are getting from the AIC function is wrong when you are comparing transformations of the "dependent variable". The point is $-2\log (p (y|\theta)) $ changes under the transformation, $ x=g (y) $ (for eg,$ x=log (y) $). You have to account for the jacobian of this change when using AIC. The AIC() function you are using does not account for this. $\endgroup$ – probabilityislogic Jun 1 '14 at 5:45
  • $\begingroup$ @probabilityislogic: Do you have any academic references for your suggestion (AIC (uu0)+2*sum (log (usili))) so that I can cite them in academic writings? Thanks. $\endgroup$ – KuJ Sep 11 '14 at 8:52
3
$\begingroup$

Taken from Akaike 1978

This excerpt from Akaike 1978 provides a citation in support of the solution by @probabilityislogic.

Akaike, H. 1978. On the Likelihood of a Time Series Model. Journal of the Royal Statistical Society. Series D (The Statistician) 27:217-235.

$\endgroup$
  • 1
    $\begingroup$ sorry I don't understand, what is "transformation of a variable" and how is it related to my question. Please explain, thanks $\endgroup$ – Curious Dec 13 '16 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.