Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work?

I just came around this question when I did comparison like this:

> uu0 = lm(log(usili) ~ rok)
> uu1 = lm(usili ~ rok)
> AIC(uu0)
[1] 3192.14
> AIC(uu1)
[1] 14277.29


This way I justified the log transformation of variable usili. But I don't know if I can AIC-compare models when for example the dependent variable is different?

Ideal answer would include the list of prerequisites (mathematical assumptions).

• There are lots of prerequisites. In practice, one prerequisite that frequently leads to wrong answers occurs when the sample size is too small for AIC, or BIC to yeild probably correct answers. Note, AIC, and BIC are methods that are only asymptotically correct. See this answer, stats.stackexchange.com/a/376064/99274
– Carl
Mar 14, 2021 at 5:35

You can not compare the two models as they do not model the same variable (as you correctly recognise yourself). Nevertheless AIC should work when comparing both nested and nonnested models.

Just a reminder before we continue: a Gaussian log-likelihood is given by

$$\log(L(\theta)) =-\frac{|D|}{2}\log(2\pi) -\frac{1}{2} \log(|K|) -\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu),$$

$K$ being the covariance structure of your model, $|D|$ the number of points in your datasets, $\mu$ the mean response and $x$ your dependent variable.

More specifically AIC is calculated to be equal to $2k - 2 \log(L)$, where $k$ is the number of fixed effects in your model and $L$ your likelihood function [1]. It practically compares trade-off between variance ($2k$) and bias ($2\log(L)$) in your modelling assumptions. As such in your case it would compare two different log-likelihood structures when it came to the bias term. That is because when you calculate your log-likelihood practically you look at two terms: a fit term, denoted by $-\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu)$, and a complexity penalization term, denoted by $-\frac{1}{2} \log(|K|)$. Therefore you see that your fit term is completely different between the two models; in the first case you compare the residuals from the raw data and in the other case the residuals of the logged data.

Aside Wikipedia, AIC is also defined to equate: $|D| \log\left(\frac{RSS}{|D|}\right) + 2k$ [3]; this form makes it even more obvious why different models with different dependent variable are not comparable. The RSS is the two case is just incomparable between the two.

Akaike's original paper [4] is actually quite hard to grasp (I think). It is based on KL divergence (difference between two distributions roughly speaking) and works its way on proving how you can approximate the unknown true distribution of your data and compare that to the distribution of the data your model assumes. That's why "smaller AIC score is better"; you are closer to the approximate true distribution of your data.

So to bring it all together the obvious things to remember when using AIC are three [2,5] :

1. You can not use it to compare models of different data sets.

2. You should use the same response variables for all the candidate models.

3. You should have $|D| >> k$, because otherwise you do not get good asymptotic consistency.

Sorry to break the bad news to you but using AIC to show you are choosing one dependent variable over another is not a statistically sound thing to do. Check the distribution of your residuals in both models, if the logged data case has normally distributed residuals and the raw data case doesn't, you have all the justification you might ever need. You might also want to check if your raw data correspond to a lognormal, that might be enough of a justification also.

For strict mathematical assumptions the game is KL divergence and information theory...

Ah, and some references:

1. http://en.wikipedia.org/wiki/Akaike_information_criterion
2. Akaike Information Criterion, Shuhua Hu, (Presentation p.17-18)
3. Applied Multivariate Statistical Analysis, Johnson & Wichern, 6th Ed. (p. 386-387)
4. A new look at the statistical model identification, H. Akaike, IEEE Transactions on Automatic Control 19 (6): 716–723 (1974)
5. Model Selection Tutorial #1: Akaike’s Information Criterion, D. Schmidt and E. Makalic, (Presentation p.39)
• thanks! I did not understood the math but I got the core of the message. However, can you please list all the prerequisites needed for AIC model comparison? Just to be sure I will not make another mistake next time. I'll go and check them one by one. Jan 28, 2013 at 22:16
• I am afraid I don't have a "check-list" as such. Ref.[2] has a quite comprehensive list if you are interested though. Main things to remember are that: 1. because AIC is an asymptotically efficient model selection criterion you need $|D|$ to be significantly greater than $p$ and 2. you can to use it only to compare models of the same dependent data. Mathematically speaking you want $L(\theta)$ to be twice differentiable, every candidate model $\theta$ to be mapped a unique $p(x|\theta)$ and your ML estimates to be consistent, but I think these assumptions are an overkill to show in a paper... Jan 28, 2013 at 22:45
• thank you for adding list of those 3 assumptions to the answer! That's what I needed. Jan 29, 2013 at 9:04
• Looking at your answer again: your point 1. "You can not use it to compare models of different data sets". What you mean by "data set"? What if I change the set of dependent variables? I guess that in that case AIC should be still comparable? Can you please update your answer to clarify this? Jul 24, 2013 at 7:11
• (Sorry for the very late reply!) I think you want to say independent variables... If you change your dependent variable you are messing up with your $RSS$ once more as the "model fits" (roughly speaking, $\mu$) are not compared against the same $x$. (Take your time answering @Curious, I won't expecting anything before mid July! :D ) Jan 23, 2014 at 23:56

You should be able to compare using AIC in principle, just that the number called "AIC" is not the number you need. You are comparing normal vs log-normal distributions. Now the AIC from model uu0 is basically just missing the "jacobian" of the log transformation. For a log normal model, this is simply $\prod_i y_i^{-1}$. To convert this to AIC you need to take negative twice log of this term, which means that you need to add $2\sum_i\log (y_i)$ to the AIC number for uu0. So you should have AIC (uu0)+2*sum (log (usili)) being compared with AIC (uu1)

• I don't understand what you follow with your attempt to "correct" AIC somehow and what did you actually get by it (how to interpret your result). Anyway, don't dig into this, it doesn't matter because my question was about something completely different: what are the general prerequisites for the AIC (actual, uncorrected) to be sensibly comparable. Don't focus on this particular example, it's just an example of the general thing. May 31, 2014 at 16:03
• @curious - my point is that my "corrected AIC" is the actual AIC, and the thing you are getting from the AIC function is wrong when you are comparing transformations of the "dependent variable". The point is $-2\log (p (y|\theta))$ changes under the transformation, $x=g (y)$ (for eg,$x=log (y)$). You have to account for the jacobian of this change when using AIC. The AIC() function you are using does not account for this. Jun 1, 2014 at 5:45
• @probabilityislogic: Do you have any academic references for your suggestion (AIC (uu0)+2*sum (log (usili))) so that I can cite them in academic writings? Thanks.
– KuJ
Sep 11, 2014 at 8:52
• @KuJ Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). Springer-Verlag. doi.org/10.1007/b97636, section 2.11.3 Oct 7, 2021 at 18:04

Transformation of Variable
In the analysis of time series it is common to try some kind of transformation on the variable. The decision on the choice of the transformation can be realized very simply by using the likelihoods of the models. Figure 4 illustrates the sunspot series for the years 1749-1924. For the original series, $$y(n)$$, the two transformations $$\left\{y(n) + 1\right\}^{1/2}$$ and $$\log\left\{y(n) + 1\right\}$$ were applied. The choice of the additive constant 1 was quite arbitrary and the results were intended only for the purpose of illustration.

AIC attains the minimum values at $$\text{ARMA}(7,2)$$, for the original $$y(n)$$, $$\text{ARMA}(7,3)$$ for $$\left\{y(n) + 1\right\}^{1/2}$$ and $$\text{ARMA}(5,4)$$ for $$\log\left\{y(n) + 1\right\}$$. The effect of transforming the variable is represented simply by the multiplication of the likelihood by the corresponding Jacobian and thus by the addition of minus twice the logarithm of the Jacobian to the AIC. For the case of $$\left\{y(n) + 1\right\}^{1/2}$$, this last quantity is equal to $$\sum \log\left\{y(n) + 1\right\} + 2N\log2$$ and, for the case of $$\log\left\{y(n) + 1\right\}$$, it is $$2\sum \log\left\{y(n) + 1\right\}$$, where the summation extends over $$n = 1, 2, \ldots, N$$.

This excerpt from Akaike 1978 provides a citation in support of the solution by @probabilityislogic.

Akaike, H. 1978. On the Likelihood of a Time Series Model. Journal of the Royal Statistical Society. Series D (The Statistician) 27:217-235.

• sorry I don't understand, what is "transformation of a variable" and how is it related to my question. Please explain, thanks Dec 13, 2016 at 10:12
• @Tomas Very simply, log(usili) is a transformation of usili. The point of the answers by probabilityislogic and bjd is that whenever the only difference between the response variables is that one [log(usilii)] is a mere transformation of the other [usili], then you can still validly apply and compare AICs as long as you transform the AIC values as well as they have described. Jun 28, 2023 at 23:02