Prerequisites for AIC model comparison

Question

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work?

I just came around this question when I did comparison like this:

> uu0 = lm(log(usili) ~ rok)
> uu1 = lm(usili ~ rok)
> AIC(uu0)
[1] 3192.14
> AIC(uu1)
[1] 14277.29

This way I justified the log transformation of variable usili. But I don't know if I can AIC-compare models when for example the dependent variable is different?

Ideal answer would include the list of prerequisites (mathematical assumptions).

Answer 1

You can not compare the two models as they do not model the same variable (as you correctly recognise yourself). Nevertheless AIC should work when comparing both nested and nonnested models.

Just a reminder before we continue: a Gaussian log-likelihood is given by

$$ \log(L(\theta)) =-\frac{|D|}{2}\log(2\pi) -\frac{1}{2} \log(|K|) -\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu), $$

$K$ being the covariance structure of your model, $|D|$ the number of points in your datasets, $\mu$ the mean response and $x$ your dependent variable.

More specifically AIC is calculated to be equal to $2k - 2 \log(L)$, where $k$ is the number of fixed effects in your model and $L$ your likelihood function [1]. It practically compares trade-off between variance ($2k$) and bias ($2\log(L)$) in your modelling assumptions. As such in your case it would compare two different log-likelihood structures when it came to the bias term. That is because when you calculate your log-likelihood practically you look at two terms: a fit term, denoted by $-\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu)$, and a complexity penalization term, denoted by $-\frac{1}{2} \log(|K|)$. Therefore you see that your fit term is completely different between the two models; in the first case you compare the residuals from the raw data and in the other case the residuals of the logged data.

Aside Wikipedia, AIC is also defined to equate: $|D| \log\left(\frac{RSS}{|D|}\right) + 2k$ [3]; this form makes it even more obvious why different models with different dependent variable are not comparable. The RSS is the two case is just incomparable between the two.

Akaike's original paper [4] is actually quite hard to grasp (I think). It is based on KL divergence (difference between two distributions roughly speaking) and works its way on proving how you can approximate the unknown true distribution of your data and compare that to the distribution of the data your model assumes. That's why "smaller AIC score is better"; you are closer to the approximate true distribution of your data.

So to bring it all together the obvious things to remember when using AIC are three [2,5] :

1. You can not use it to compare models of different data sets.

2. You should use the same response variables for all the candidate models.

3. You should have $|D| >> k$, because otherwise you do not get good asymptotic consistency.

Sorry to break the bad news to you but using AIC to show you are choosing one dependent variable over another is not a statistically sound thing to do. Check the distribution of your residuals in both models, if the logged data case has normally distributed residuals and the raw data case doesn't, you have all the justification you might ever need. You might also want to check if your raw data correspond to a lognormal, that might be enough of a justification also.

For strict mathematical assumptions the game is KL divergence and information theory...

Ah, and some references:

1. http://en.wikipedia.org/wiki/Akaike_information_criterion
2. Akaike Information Criterion, Shuhua Hu, (Presentation p.17-18)
3. Applied Multivariate Statistical Analysis, Johnson & Wichern, 6th Ed. (p. 386-387)
4. A new look at the statistical model identification, H. Akaike, IEEE Transactions on Automatic Control 19 (6): 716–723 (1974)
5. Model Selection Tutorial #1: Akaike’s Information Criterion, D. Schmidt and E. Makalic, (Presentation p.39)

Answer 2

You should be able to compare using AIC in principle, just that the number called "AIC" is not the number you need. You are comparing normal vs log-normal distributions. Now the AIC from model uu0 is basically just missing the "jacobian" of the log transformation. For a log normal model, this is simply $\prod_i y_i^{-1} $. To convert this to AIC you need to take negative twice log of this term, which means that you need to add $2\sum_i\log (y_i)$ to the AIC number for uu0. So you should have AIC (uu0)+2*sum (log (usili)) being compared with AIC (uu1)

Answer 3

This excerpt from Akaike 1978 provides a citation in support of the solution by @probabilityislogic.

Akaike, H. 1978. On the Likelihood of a Time Series Model. Journal of the Royal Statistical Society. Series D (The Statistician) 27:217-235.