13
$\begingroup$

In parameter estimation, it's common to report a 95% CI around each parameter. Why don't I see AIC (or deltaAIC) with a CI?

If I bootstrap the fitting of two potential models, and get a deltaAIC for each iteration, would reporting a 95% CI be meaningful?

Edit: I don't mean to imply that AIC is the same as parameter estimation. I'm asking why we treat goodness-of-fit estimates (AIC, BIC, etc.) differently from estimates that are reported with a CI.

$\endgroup$
6
  • 3
    $\begingroup$ What would you need the confidence intervals for? $\endgroup$
    – Tim
    Jun 5, 2021 at 18:17
  • 2
    $\begingroup$ @Tim Let's say a point estimate for deltaAIC is 2.5, it might seem like a moderate difference. But if the 95%CI is [-1, 4], the result is too noisy for there to be a reliable difference between the two models. However, if the CI is [2, 3], the difference is more clear. $\endgroup$
    – sharoz
    Jun 5, 2021 at 18:21
  • 1
    $\begingroup$ @Tim, in addition to sharoz's valid response, see also my answer for an idea. $\endgroup$ Jun 6, 2021 at 10:33
  • 1
    $\begingroup$ @MatchMakerEE, what about the proof that AIC is the efficient model selector (compare to another proof that BIC is consistent but not efficient)? Does it contain a mistake? If not, why should one use another criterion if the goal is forecast accuracy? $\endgroup$ Jun 6, 2021 at 10:55
  • 1
    $\begingroup$ @sharoz: I added to my answer in response to your edit. $\endgroup$ Jun 7, 2021 at 8:45

2 Answers 2

14
$\begingroup$

AIC estimates $-2n \ \times$ the expected likelihood on a new, unseen data point from the data generating process (DGP) that generated your sample.* Even though the target (the estimand) is not a parameter, it is a meaningful quantity. E.g. it may be interpreted as the expected loss of a point prediction. It is quite natural to wish for a confidence interval around the point estimate (the AIC). This way we can tell not only what the expected loss is but also how uncertain it is. In summary, while I do not have a ready answer for how to obtain the confidence interval and under what conditions your idea of bootstrapping may work, I clearly do see a point in pursuing it.

*See How can we select the best GARCH model by carrying out likelihood ratio test?, Can results for model selection with AIC be interpretable at the population level?, Using AIC/BIC within cross-validation for likelihood based loss functions among other threeads where this idea is employed.

$\endgroup$
5
  • 1
    $\begingroup$ I'd like to emphasize this by saying that in my application field, I often have structure in my data that I cannot adequately convey to the model I train. In many cases I can get sensible predictive quality, but any error or quality (incl. goodness of fit) estimates from within that model may be way off. But I may be able to set up a bootstrap in a way that reflects this structure. $\endgroup$ Jun 6, 2021 at 11:47
  • $\begingroup$ @Richard Hardy: This is a fair point and I have amended my answer to take this into account. $\endgroup$ Jun 6, 2021 at 12:56
  • $\begingroup$ @Lewian, thanks, your edit makes sense to me. I also phrased my answer carefully with yours in mind. Mine does not (or at least is not intended to) contradict yours, it is just another perspective. $\endgroup$ Jun 6, 2021 at 13:56
  • $\begingroup$ @RichardHardy Thanks. Looking at your answer for "Can results...": As n->∞, AIC behaves similarly to loglik. That leads to a follow-up: why not report loglik with a CI? $\endgroup$
    – sharoz
    Jun 7, 2021 at 2:45
  • 1
    $\begingroup$ @sharoz, similarly to the case of AIC, I do not see anything conceptually wrong with reporting a confidence interval for log-likelihood. On the other hand, perhaps only a (very) small fraction of users are typically interested in the confidence interval for log-likelihood, so it is not reported for the sake of brevity. $\endgroup$ Jun 7, 2021 at 6:22
22
$\begingroup$

The AIC is not an estimator of a true parameter. It is a data-dependent measurement of the model fit. The model fit is what it is, there is no model fit that is any "truer" than the one you have, because it's the one you have that is measured. But without any true parameter for which the AIC would be an estimator, one cannot have a confidence interval (CI).

I'm by the way not disputing the answer by Richard Hardy. The AIC, as some other quantities such as $R^2$, can be interpreted as estimating something "true but unobservable", in which case one can argue that a CI makes sense. Personally I find the interpretation as measuring fit quality more intuitive and direct, for which one wouldn't have a CI for the reasons above, but I'm not saying that there is no way for it to be well defined and of some use.

Edit: As a response to the addition in the question: "I don't mean to imply that AIC is the same as parameter estimation. I'm asking why we treat goodness-of-fit estimates (AIC, BIC, etc.) differently from estimates that are reported with a CI." - The definition of a CI relies on a parameter being estimated. It says that given the true parameter value the CI catches this value with probability $(1-\alpha)$. As long as you're not interested in that true parameter value, a CI is meaningless.

$\endgroup$
12
  • 6
    $\begingroup$ +1 but it might help to explain why this is in contrast to something like MSE or $R^2$. $\endgroup$
    – Dave
    Jun 5, 2021 at 20:24
  • 1
    $\begingroup$ @Dave: You can do that if you think it makes sense; I hardly ever see CIs for $R^2$ and MSEs. From my point of view the meaning of these is less intuitive than the fact that there's none for AIC. $\endgroup$ Jun 5, 2021 at 20:37
  • $\begingroup$ MSE, for instance (and depending on the denominator you use), estimates the standard deviation of the error term, which is a "true" parameter. $\endgroup$
    – Dave
    Jun 5, 2021 at 20:40
  • 3
    $\begingroup$ @MichaelM, for AIC the population analog is twice the negative average log-likelihood. If you find it easy to see the population analog of MSE (average squared error), I think it is not that difficult to see it for AIC; but perhaps it is a subjective impression. $\endgroup$ Jun 6, 2021 at 14:02
  • 1
    $\begingroup$ @sharoz: There are techniques designed to identify and deal with outliers that are much better than that. $\endgroup$ Jun 7, 2021 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.