7
$\begingroup$

Let's assume we have two linear regression models $m_1$ and $m_2$, where $m_2$ is nested in $m_1$, and two data sets $d_1$ and $d_2$ which are of different size.

Calculating the AIC for each pair shows that the following equation is true:

$$ AIC_{m_2,d_1} - AIC_{m_1,d_1} > AIC_{m_2,d_2} - AIC_{m_1,d_2} $$

Can we conclude from this equation that the non-nested/full model $m_1$ "improves" $m_2$ on data $d_1$ more than on data set $d2$?

The conclusion might not be valid because the AIC values are calculated on different data sets and therefore might not be comparable. However, since actually AIC differences are compared the conclusion might be valid. What is your take on this?

Improve this question
$\endgroup$
2
  • $\begingroup$ Q: What happens when you divide the left hand side by the size of d1 and the right hand side by the size of d2? $\endgroup$
    – jwimberley
    Commented Dec 20, 2016 at 14:48
  • 1
    $\begingroup$ No. AIC is specified to a given set of data, it is conceptually wrong to draw conclusion among datasets. A way you can "order"(compare them if you really want) models among different datasets is to use Kullback-Leibler divergence, a reference is Perlman(1983)The limiting behavior of multiple roots of the likelihood equation. $\endgroup$
    – Henry.L
    Commented Dec 20, 2016 at 16:08

1 Answer 1

Reset to default
1
+50
$\begingroup$

The AIC criterion scales with the overall size of the dataset, and this is true for differences in AIC values as well. The criterion is based on the relationship $$ -2 \, \mathrm{E}[\log \mathrm{Pr}_{\hat \theta}(Y)] \approx -\frac{2}{N} \, \mathrm{E}[\mathrm{loglik}] + \frac{2d}{N} $$ where $d$ is the number of parameters in the likelihood function being maximized (Elements of Statistical Learning equation 7.27). The term on the left is the expected out-of-sample "error" rate, using the log of the probability as the error metric. The right hand consists of the in-sample error rate estimated from the maximized log-likelihood, plus the term $2d/N$ correcting for the optimism of the maximized log-likelihood. The most important factor here is the $N$ in the denominator of the right hand side. The AIC is typically defined as $$ \mathrm{AIC} = -2 \, \mathrm{loglik} + 2d $$ (although the ESL textbook adds a $1/N$ factor). In this form, the AIC predicts $N$ times the out-of-sample error rate. To compare AIC differences from two samples, you should divide the AIC values by the sample size to compare them on equal terms.

Improve this answer
$\endgroup$
12
  • $\begingroup$ I disagree, dividing by sample size will not solve any problem. Consider the gaussian case where AIC is equivalent to F-tests for nested models, adjust for sample size is proven to be problematic as adjusted R^2. Read my comments above. $\endgroup$
    – Henry.L
    Commented Dec 20, 2016 at 16:10
  • 1
    $\begingroup$ @CagdasOzgenc You probably misunderstand my point. I do not think the question is even relevant to sample size. The point is you cannot compare two models fitted to different data sets using any information criterion. If my memory is correct, Christopher has discussed this question in his IMS monograph(small sample asymptotics). Thanks for the input and happy holiday! $\endgroup$
    – Henry.L
    Commented Dec 24, 2016 at 13:41
  • 1
    $\begingroup$ @Henry.L I cannot attest for conditional distributions, but for unconditional distributions I claim that you can work on two different sample sets (and possibly with different lengths) generated by the same true data process provided that the sample size is large. In each case AIC/sample size will yield expected cross entropy, hence they are comparable IMHO. $\endgroup$
    – Cagdas Ozgenc
    Commented Dec 24, 2016 at 19:54
  • 2
    $\begingroup$ @CagdasOzgenc "...generated by the same true data process provided that the sample size is large." The problem in reality is that you never know whether two datasets are actually generated from the same model/process. Even if they are, the adjustment is no more than a rescale and hence does not address the problem raised by the OP since he stated very clearly that two models are nested. I hope it is clear to you now. It is just very wrong to do so because ESL said so, "machine learning" provides many results that is only valid in specific cases. $\endgroup$
    – Henry.L
    Commented Dec 24, 2016 at 20:02
  • 2
    $\begingroup$ @Henry.L According to your logic we shouldn't do cross-validation studies because when we split a data set into two subsets (one for training one for validation) they may god forbid (pun intended) come from different processes. $\endgroup$
    – Cagdas Ozgenc
    Commented Dec 24, 2016 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.