# What is the actual significance of a difference in AIC or BIC values?

Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, especially, but not only when non-nested models are compared, we use the lower value of the AIC, AICc, BIC or other information content index to suggest what the better model is. However, more generally, entropy is case-wise, i.e., data-wise, variable.

Question With what certainty do we know, based on comparative information content indices from a particular data set, that that lower index value properly suggests the correct model more generally for a less limited data set?

I feel that non-nested model comparison of information content is not always relevant in all circumstances, for example see this Q/A. Nesting is when all of the models tested can be derived by eliminating parameters from a parent model. Non-nesting is when the models contain parameters that are not in a set with subset(s) format.

I really would appreciate any insight into the variability of comparison of information content for either nested or non-nested models in the context of subset data.

• Same problem here: stats.stackexchange.com/questions/361065/… Oct 10 '18 at 20:37
• @Alex I saw that question and answer. The person answering was not entirely satisfied with the answer given, and although the discussion is somewhat related to the topic here, it is not as focused. For example, the mechanics of what AIC/BIC etc. are extracting from the data are not explored, and that is the focus here.
– Carl
Oct 10 '18 at 20:46

The difference in AIC (or BIC) for two models is twice the log-likelihood ratio minus a constant: it follows immediately that in any particular case selecting the AIC corresponds to performing a likelihood-ratio test, but that in different cases it corresponds to tests of different significance levels.

With nested models, the null hypothesis has to be that the smaller model holds. Given some regularity conditions, Wilks' theorem applies; so if $$p$$ is the difference in the number of free parameters between the models, asymptotically the probability of AIC's selecting the larger model when the smaller one in fact holds is the probability that a chi-squared r.v. with $$p$$ degrees of freedom exceeds $$2p$$. For $$p=1$$ the significance of the test is 0.157; for $$p=2$$, 0.135; & so on. When exact tests are possible the distribution of the log-likelihood ratio of course depends on precisely what the models are.

With non-nested models, even finding an asymptotic distribution for the log-likelihood ratio involves the calculation of rather complicated expectations under the null (see Cox's or Vuong's papers referenced in Generalized log likelihood ratio test for non-nested models & Comparison of log-likelihood of two non-nested models). I doubt much can be said in general about the significance of a difference in AIC.

The moral has already been given, pithily, by @RichardHardy:

How do you define what is a "correct model"? Is it the data generating process (DGP)? If so, why would you be using AIC trying to identify the DGP? The question AIC is answering is not "Which of the models is the DGP?". Try asking a different question, such as "Which model will give better predictions under a certain type of loss (associated with the likelihood being used)?", and you might find that AIC is answering "correctly" (or perhaps not?). That is, use a hammer for hammering nails

Problems with the AIC include its accuracy in small samples (the bias-correction term is only to first order) & when neither model is particularly close (in the sense of Kullback–Leibler divergence) to the true model; but it can't fairly be criticized for not doing what it wasn't made for: there are hypothesis tests for that.

• Shucks, I wanted to award you the bounty, so as to not waste the points. I thought that would happen automatically, but, it didn't.
– Carl
Oct 18 '18 at 0:18
• However, I can upvote your answer, not because I believe it (or disbelieve it), but because it contributes substantially to the discussion. (+1).
– Carl
Oct 18 '18 at 3:28
• From Information Criterion for Minimum Cross-Entropy Model Selection "To simplify the model complexity penalty term in the AIC, Akaike (1974) makes the strong assumption that the true distribution of data belongs to the parametric distribution family being considered." Thus, AIC assumes that the correct model is being considered. Asking me to define it is besides the point, it is already assumed. Rather, to defend AIC as correct, the burden of proof is on its proponents.
– Carl
Oct 18 '18 at 17:16
• Moreover, I do not see how in completely non-nested models, like ND and UD, one can make a correct model assumption from "within a parametric distribution family." Once again, the burden of proof for non-nesting case is on its proponents. All I have shown in my answer is that "squaring the circle" does not necessarily have a "Happy ending."
– Carl
Oct 18 '18 at 17:28
• @Carl: There's something to that, as K-L divergence is hardly the only way to quantify how far a fitted model is from the truth - I probably wouldn't have heard of it if, per impossible, it weren't what's minimized by ML estimation. Nevertheless, even when you know the true model for a fact, bar p parameters, there's a question remaining of whether, when you estimate those parameters from a particular data-set, you'll end up with a fit that's closer than if you estimate the parameters of another, simpler, model; & that's the question AIC's designed to answer. Oct 19 '18 at 16:02

Suppose one generates values from a standard normal distribution, $$\mathcal{N}(0,1)$$. If we have only generated two values, $$n=2$$, then we have a discrete uniform distribution, not a convincingly discrete approximation of a normal distribution. Indeed, this is true for any $$n=2$$, no matter which generating distribution gave rise to those values, a discrete uniform distribution is a default result. Normal and uniform distributions are non-nested with respected to each other. Indeed, they have very different shapes. If we generate $$\mathcal{N}(0,1)$$ for increasing $$n$$ and examine AIC for fitting with a normal distribution versus a uniform distribution, even though we know that our generating function is $$\mathcal{N}(0,1)$$, AIC will not always be lesser for a normal distribution fit than for a uniform distribution fit. The plot below shows how many times out of 1000 repetitions AIC for a normal distribution model was better (less than) AIC for a uniform distribution model for $$n$$ varying from $$n=5$$ to $$n=100$$.

As can be seen in the image, AIC for a normal distribution (i.e., the correct answer) was only selected to be better than a uniform distribution 395 times out of 1000 trials or 39.5% of the time for $$n=5$$. This increased to 949 times out of 1000 trials for $$n=100$$, a value still having an error rate of slightly more than 5%. It is said that AIC is asymptotically correct, and that appears to be correct. BTW, BIC makes the same choices for both 2 parameter models as AIC. But is that useful for small to moderately sized values of $$n$$?

Above is an example of observed probability of model selection. It is claimed that the likelihood of AIC choosing a correct model is as follows:

The quantity $$\exp\frac{\text{AIC}_{min} − \text{AIC}_i}{2}$$ is known as the relative likelihood of model $$i$$. It is closely related to the likelihood ratio used in the likelihood-ratio test. Indeed, if all the models in the candidate set have the same number of parameters, then using AIC might at first appear to be very similar to using the likelihood-ratio test. There are, however, important distinctions. In particular, the likelihood-ratio test is valid only for nested models, whereas AIC (and AICc) has no such restriction.

Now note that the likelihood above can be reciprocated. That is, if model A is twice as likely as model B, then model B is one-half as likely as model A. In the current context, we are not dealing with likelihoods, we created a Monte Carlo simulation with truth data, such that we observed the probability of making the correct choices. We have observed in this simulation that the likelihood of making the correct choice is heavily influenced by $$n$$, the number of observations, and that unless $$n$$ is large, we did not seem to get reliable answers.

About the program: lists are initialized as normal distribution (nd) AIC (ndAlist), nd BIC (ndBlist), uniform distribution (ud) AIC and BIC (udAlist, udBlist). Two do loops are used. The outer do loop increments $$n$$ from 5 to 100 in increments of $$n=5$$. The inner do loop (1) creates $$n$$ random variates (named dat) from an $$\mathcal{N}(0,1)$$. Then (2) creates an emperical CDF named edistdata from dat. (3) Defines cdfn and cdfu functions for fitting from the CDFs of nd and ud. (4) Best fits by variation of parameters of cdfn and cdfu to edistdata. Note: Fitting to CDFs rather than PDFs markedly decreases noise and is a common procedure. This is done, rather than, for example, using mean and variance to calculate nd or min and max to calculate ud because the fitting uses a single algorithm for both nd and ud and that NonlinearModelFit routine outputs AIC and BIC for the models as well as parameters as options for the output nlmn and nlmu fit outputs, e.g., as nlmn["AIC"]. Note: it is assumed that the AIC and BIC fit parameters are correctly calculated using ML as the contrary case would be meaningless.

(*Mathematica Program*)
ndAlist = {};
ndBlist = {};
udAlist = {};
udBlist = {};
Do[
AICndlist = {};
BICndlist = {};
AICudlist = {};
BICudlist = {};
Do[dat =
RandomVariate[NormalDistribution[0, 1], n, WorkingPrecision -> 40];
edistdata = Table[{x, CDF[EmpiricalDistribution[dat], x]}, {x, dat}];
cdfn[a1_, a2_, x_] := CDF[NormalDistribution[a1, a2], x];
cdfu[b1_, b2_, x_] := CDF[UniformDistribution[{b1, b2}], x];
nlmn = NonlinearModelFit[edistdata, cdfn[a1, a2, x], {{a1, 0}, {a2, 1}}, x];
nlmu = NonlinearModelFit[edistdata, cdfu[b1, b2, x], {{b1, -2}, {b2, 2}}, x];
AICndlist = AppendTo[AICndlist, nlmn["AIC"]];
BICndlist = AppendTo[BICndlist, nlmn["BIC"]];
AICudlist = AppendTo[AICudlist, nlmu["AIC"]];
BICudlist = AppendTo[BICudlist, nlmu["BIC"]],
{i, 1, 1000}];
ndA = 0.; udA = 0.; ndB = 0.; udB = 0.;
Do[If[AICndlist[[j]] < AICudlist[[j]], ndA = ndA + 1, udA = udA + 1],
{j, 1, 1000}];
Do[If[BICndlist[[j]] < BICudlist[[j]], ndB = ndB + 1, udB = udB + 1],
{j, 1, 1000}];
Print["n: ", n, "\nAIC nd/1000: ", ndA, "\tAIC ud/1000: ", udA, "\nBIC nd/1000: ", ndB, "\tBIC ud/1000: ", udB];
ndAlist = AppendTo[ndAlist, {n, ndB}];
ndBlist = AppendTo[ndBlist, {n, ndB}], {n, 5, 100, 5}]
Print[ndAlist]
ListPlot[ndAlist, AxesLabel -> {"n", "AIC ND < AIC UD"}, PlotRange -> {{0, 100}, {0, 1000}}, PlotRangePadding -> {{0, 1}, {0, 0}}]


(Numerical Output)

n: 5
AIC nd/1000: 395.   AIC ud/1000: 605.
BIC nd/1000: 395.   BIC ud/1000: 605.

n: 10
AIC nd/1000: 572.   AIC ud/1000: 428.
BIC nd/1000: 572.   BIC ud/1000: 428.

n: 15
AIC nd/1000: 684.   AIC ud/1000: 316.
BIC nd/1000: 684.   BIC ud/1000: 316.

n: 20
AIC nd/1000: 725.   AIC ud/1000: 275.
BIC nd/1000: 725.   BIC ud/1000: 275.

n: 25
AIC nd/1000: 769.   AIC ud/1000: 231.
BIC nd/1000: 769.   BIC ud/1000: 231.

n: 30
AIC nd/1000: 777.   AIC ud/1000: 223.
BIC nd/1000: 777.   BIC ud/1000: 223.

n: 35
AIC nd/1000: 811.   AIC ud/1000: 189.
BIC nd/1000: 811.   BIC ud/1000: 189.

n: 40
AIC nd/1000: 841.   AIC ud/1000: 159.
BIC nd/1000: 841.   BIC ud/1000: 159.

n: 45
AIC nd/1000: 848.   AIC ud/1000: 152.
BIC nd/1000: 848.   BIC ud/1000: 152.

n: 50
AIC nd/1000: 848.   AIC ud/1000: 152.
BIC nd/1000: 848.   BIC ud/1000: 152.

n: 55
AIC nd/1000: 877.   AIC ud/1000: 123.
BIC nd/1000: 877.   BIC ud/1000: 123.

n: 60
AIC nd/1000: 886.   AIC ud/1000: 114.
BIC nd/1000: 886.   BIC ud/1000: 114.

n: 65
AIC nd/1000: 900.   AIC ud/1000: 100.
BIC nd/1000: 900.   BIC ud/1000: 100.

n: 70
AIC nd/1000: 901.   AIC ud/1000: 99.
BIC nd/1000: 901.   BIC ud/1000: 99.

n: 75
AIC nd/1000: 914.   AIC ud/1000: 86.
BIC nd/1000: 914.   BIC ud/1000: 86.

n: 80
AIC nd/1000: 932.   AIC ud/1000: 68.
BIC nd/1000: 932.   BIC ud/1000: 68.

n: 85
AIC nd/1000: 935.   AIC ud/1000: 65.
BIC nd/1000: 935.   BIC ud/1000: 65.

n: 90
AIC nd/1000: 946.   AIC ud/1000: 54.
BIC nd/1000: 946.   BIC ud/1000: 54.

n: 95
AIC nd/1000: 952.   AIC ud/1000: 48.
BIC nd/1000: 952.   BIC ud/1000: 48.

n: 100
AIC nd/1000: 949.   AIC ud/1000: 51.
BIC nd/1000: 949.   BIC ud/1000: 51.

{{5,395.},{10,572.},{15,684.},{20,725.},{25,769.},{30,777.},{35,811.},{40,841.},{45,848.},{50,848.},{55,877.},{60,886.},{65,900.},{70,901.},{75,914.},{80,932.},{85,935.},{90,946.},{95,952.},{100,949.}}


Plot of output was shown above.

The answer here echos the results of a recent paper by Yafune et al. A Note on Sample Size Determination for Akaike Information Criterion (AIC) Approach to Clinical Data Analysis, which unfortunately is behind a paywall. Those authors state in their discussion that "AIC is generally used without paying attention to the probabilities corresponding to the power of statistical tests. Since AIC is usually used for exploratory analysis, it is often difficult to determine the sample sizes in advance. For such cases, it is desirable to investigate afterwards whether the sample sizes are large enough by checking the probabilities corresponding to the power of statistical tests. If the sample sizes are not large enough, it is possible that the AIC approach does not lead us to the conclusions which we seek."

To this we would only add that the sample sizes indicated in that paper can easily exceed 100 for a power of 0.8.

• An interesting tread. I am not sure if my comment is relevant here, but let me try. How do you define what is a "correct model"? Is it the data generating process (DGP)? If so, why would you be using AIC trying to identify the DGP? The question AIC is answering is not "Which of the models is the DGP?". Try asking a different question, such as "Which model will give better predictions under a certain type of loss (associated with the likelihood being used)?", and you might find that AIC is answering "correctly" (or perhaps not?). That is, use a hammer for hammering nails. Oct 11 '18 at 13:51
• Recall that AIC's used to compare pre-specified models with unknown parameters fit to observations by maximum likelihood. If I've read this right, you're defining a discrete uniform distribution on a support determined by the observations. Oct 13 '18 at 9:35
• There are some regularity conditions that determine whether AIC has various properties or not (please don't ask me about them!); but AIC is defined as twice the negative log-likelihood at the maximum-likelihood parameter estimates plus twice the number of free parameters estimated, so your simulation has to be be of fitting two different models to the same data by maximum likelihood, else it makes no sense. For example, you could fit the rate of an exponential distribution, & fit the log-location & log-scale of a log-normal distribution. Oct 13 '18 at 11:33
• What would help is explaining precisely what the models are, what their free parameters are, how you're estimating those, & how you calculate AIC when the observations are discrete under one & continuous under the other. You could illustrate the explanation with code Oct 14 '18 at 13:16
• @EngrStudent What about weights? In this case, the weights are not revealing as both models have two free parameters. Same for BIC. As $n$ is the same for both models, the choice of $n$ does not alter which BIC is the lesser value. It is the fit itself that alters the AIC/BIC selected model and there is no difference between AIC and BIC selection of the better model in this case.
– Carl
Oct 15 '18 at 19:19