# Comparing differences of AIC of different data sets

Let's assume we have two linear regression models $m_1$ and $m_2$, where $m_2$ is nested in $m_1$, and two data sets $d_1$ and $d_2$ which are of different size.

Calculating the AIC for each pair shows that the following equation is true:

$$AIC_{m_2,d_1} - AIC_{m_1,d_1} > AIC_{m_2,d_2} - AIC_{m_1,d_2}$$

Can we conclude from this equation that the non-nested/full model $m_1$ "improves" $m_2$ on data $d_1$ more than on data set $d2$?

The conclusion might not be valid because the AIC values are calculated on different data sets and therefore might not be comparable. However, since actually AIC differences are compared the conclusion might be valid. What is your take on this?

• Q: What happens when you divide the left hand side by the size of d1 and the right hand side by the size of d2? – jwimberley Dec 20 '16 at 14:48
• No. AIC is specified to a given set of data, it is conceptually wrong to draw conclusion among datasets. A way you can "order"(compare them if you really want) models among different datasets is to use Kullback-Leibler divergence, a reference is Perlman(1983)The limiting behavior of multiple roots of the likelihood equation. – Henry.L Dec 20 '16 at 16:08

The AIC criterion scales with the overall size of the dataset, and this is true for differences in AIC values as well. The criterion is based on the relationship $$-2 \, \mathrm{E}[\log \mathrm{Pr}_{\hat \theta}(Y)] \approx -\frac{2}{N} \, \mathrm{E}[\mathrm{loglik}] + \frac{2d}{N}$$ where $d$ is the number of parameters in the likelihood function being maximized (Elements of Statistical Learning equation 7.27). The term on the left is the expected out-of-sample "error" rate, using the log of the probability as the error metric. The right hand consists of the in-sample error rate estimated from the maximized log-likelihood, plus the term $2d/N$ correcting for the optimism of the maximized log-likelihood. The most important factor here is the $N$ in the denominator of the right hand side. The AIC is typically defined as $$\mathrm{AIC} = -2 \, \mathrm{loglik} + 2d$$ (although the ESL textbook adds a $1/N$ factor). In this form, the AIC predicts $N$ times the out-of-sample error rate. To compare AIC differences from two samples, you should divide the AIC values by the sample size to compare them on equal terms.