5
$\begingroup$

I am currently summarizing results from several groups of models.

Is it meaningful to report a mean AIC for each group of models? If not then how best to give a summary measure for each model group?

Some contributing factors

  • all models have the same number of parameters and structure
  • within each group, there are four models, which always have the same numbers of data points. e.g. each group has always contains four models with 50, 60, 80 and 100 data points respectively.

UPDATE

I haven't had any responses yet, but I have had further thoughts on this matter.

How do we define "meaningful"? In this case it's probably best to think in terms of what we will do with the results, the mean AIC. Interpretation of AIC is usually through likelihood function, which gives the likelihood that the worse of a pair of models is actually better (and the measured AIC difference between them is down to chance):

$\mathcal{L} = e^{-\frac{1}{2}|\Delta AIC|}$

Where $\Delta AIC$ is the difference in AIC between the models.

Without loss of generality, consider the case where we take the mean of two AICs, $\overline{AIC} = \frac{1}{2}(AIC_1 + AIC_2)$, and compare them to a bassline model which has $AIC_B=0$ while $AIC_1 > 0$ and $AIC_2 > 0$. Then we interpret the mean as a single AIC and apply the formula for $\mathcal{L}$:

$\mathcal{L}(\overline{AIC}) = e^{-\frac{1}{2}|\overline{\Delta AIC}|} = e^{-\frac{1}{2}\frac{1}{2}(AIC_1 + AIC_2)} = \sqrt{e^{-\frac{1}{2}AIC_A}e^{-\frac{1}{2}AIC_B}}$

which is the geometric mean of $\mathcal{L}_1 = \mathcal{L}(AIC_1)$ and $\mathcal{L}_2 = \mathcal{L}(AIC_2)$. So far so good: it makes sense to compute an arithmetic mean of AIC if it makes sense to compute a geometric mean of $\mathcal{L}$. (I would like to say "if and only if" but that would preclude any rationalization for doing this outside of the current argument. In the current line of thinking though this is an "iff").

So does it make sense to compute a geometric mean of $\mathcal{L}$? Presumably the reason for averaging in this manner is that we're trying to compute the expected $\mathcal{L'}$ that you will get if you apply a similar model to another dataset in future. In this case we need to look at $E(\mathcal{L'})$ the expected likelihood, assuming equal probability of each model within the group.

$E(\mathcal{L'}) = P(\mathcal{L}_1)\mathcal{L}_1 + P(\mathcal{L}_2)\mathcal{L}_2 = \frac{1}{2} \mathcal{L}_1 + \frac{1}{2} \mathcal{L}_2 = \overline{\mathcal{L}}$

This is the arithmetic, not geometric mean of $\mathcal{L}_1$ and $\mathcal{L}_2$. So no, it doesn't make sense to report the arithmetic mean of the AIC, because it doesn't make sense to report the geometric mean of $\mathcal{L}$.

Conversely, it would make sense to report the arithmetic mean of $\mathcal{L}$, so that would be a sensible way to summarize a group of models.

If anyone feels like answering it would still be useful to receive comment on the correctness (or disastrous incorrectness!) of this analysis. I am not very sure of the latter part of the argument, as $\mathcal{L}$ is a relative likelihood perhaps it is appropriate to take the geometric mean anyway?

Improve this question
$\endgroup$
6
  • $\begingroup$ If all models have the same parameters and structure why are you using AIC in the first place? What is the the difference between the models you are comparing? $\endgroup$
    – Zoë Clark
    Commented Jul 7, 2014 at 2:28
  • $\begingroup$ Oops, I don't quite mean same parameters - I mean same number of parameters representing similar things which are calculated in slightly different ways in each case. I want to use AIC rather than $r^2$ because it allows me to calculate relative likelihoods. $\endgroup$
    – Sideshow Bob
    Commented Jul 7, 2014 at 9:45
  • $\begingroup$ What is the point of combining your AICs to a single measure for each group? Could you just plot them versus the number of data points, for example? $\endgroup$
    – Evgeniy Riabenko
    Commented Jul 10, 2014 at 13:51
  • $\begingroup$ There are a lot of groups, and it's the groups I want to compare not the individual models. Reporting individual AICs per model will make it near impossible to compare groups at a glance. $\endgroup$
    – Sideshow Bob
    Commented Jul 11, 2014 at 9:13
  • $\begingroup$ Are you talking about $\Delta\mathrm{AIC}$ or $\mathrm{AIC}$? It seems like you flip back and forth in your notation. Also, are these models fitted with MLE, or by minimizing a loss function as in OLS? Yes I know they are equivalent in some cases, but it's always good to report error in terms of the actual objective function that was used. $\endgroup$
    – shadowtalker
    Commented Aug 8, 2014 at 20:46

3 Answers 3

Reset to default
2
$\begingroup$

I think that you can use median or average AIC of a group of models in comparison to another group for internal analysis or during the selection of model specifications for further consideration. There is an interpretation of the difference between AIC of two models in terms of likelihoods, see here. In a similar somewhat informal way you could compare the groups of models represented by their summary AIC.

Improve this answer
$\endgroup$
1
$\begingroup$

There is no real meaning in combining AICs. There is sometimes (seldom) benefit from combining models, e.g., Bayesian model averaging or coefficient averaging, but I haven't seen anyone combine AICs.

You did not explain why you have more than one model in the first place.

Improve this answer
$\endgroup$
7
  • $\begingroup$ In concrete terms: I am comparing different techniques for predicting vehicle flow. I try out each technique in four different areas i.e. on four different data sets. $\endgroup$
    – Sideshow Bob
    Commented Jul 8, 2014 at 16:53
  • $\begingroup$ I would like to produce a single summary statistic for each technique, to allow comparison of the effectiveness of each. Above I have referred to each technique as a "group of models". Within the group there is one model per test area. So that's why I have multiple models divided into groups. Does that make it any clearer? $\endgroup$
    – Sideshow Bob
    Commented Jul 8, 2014 at 16:54
  • $\begingroup$ What about combining relative likelihoods - is it reasonable to take a mean of these? $\endgroup$
    – Sideshow Bob
    Commented Jul 8, 2014 at 17:01
  • $\begingroup$ I think I understand the possible need for four different test datasets, but I'm not clear on why you are using different techniques as opposed to a single unified, flexible technique. $\endgroup$
    – Frank Harrell
    Commented Jul 8, 2014 at 19:48
  • $\begingroup$ When I say different techniques I mean different predictor variables derived in different ways. I am producing a comparative study to determine which of a number of simulation algorithms best predict traffic flow. Each algorithm produces a variable X - the flow in the simulation - and I regress Y, the actual measured flow, against X (in four different datasets). So I am trying to produce a single AIC for each algorithm to show its effectiveness. This would somehow be a combination of the AICs of each X-Y regression in different areas for that algorithm. $\endgroup$
    – Sideshow Bob
    Commented Jul 9, 2014 at 10:01
1
$\begingroup$

In my humble opinion, summarizing/averaging AIC/BIC makes some sense (it is quite scary to me that I'm disagreeing on this with @Frank Harrell :-). At least, I've seen this being done and written about. Please see one of my earlier answers here on Cross Validated with more information and relevant references: https://stats.stackexchange.com/a/128922/31372.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.