Akaike’s Information Criteria generally tries to find unknown model that has high dimensional reality. Bayesian Information Criteria comes across only True models.

What exactly they mean by "unknown model that has high dimensional reality" in AIC

What exactly they mean by "True models" in BIC

  • $\begingroup$ Since you put those phrases in quotes, you really should say what you are quoting from. You ask what "they mean". Who is they? $\endgroup$ – Harvey Motulsky Mar 29 '20 at 17:20
  • $\begingroup$ This is the source i got it from , what does the text in bold mean differencebetween.net/miscellaneous/… $\endgroup$ – user2478236 Mar 31 '20 at 4:44

Neither statement is quite true. The true model does not need to be in the set for the BIC to work and a model can have few dimensions and the AIC will correctly choose it.

Both are in the group of out-of-sample performance estimators called information criteria. They exist because Frequentist models assume the null is true, no matter what you use. There is no way to compare models in the original construction by Egon Pearson and Jerzy Neyman.

Information theory has been used in model selection. There are more than two information criteria. Each information criterion has different properties, from an information perspective.

Unfortunately, there is a tendency to focus on the word "Bayesian" in the BIC. All information criteria are Bayesian criterion under stylized conditions. There is a tight link between Bayesian theory and information theory. You can derive the K-L divergence from Bayes theorem. It is a transformation of the Bayesian posterior predictive density and the true density.

Because you can move back and forth between Bayesian theory and information theory, you can also make the reverse claim that all of Bayesian theory is a subset of information theory.

Although the Bayesian information criterion is directly derived from Bayesian theory, you can do the same thing with the Akaike criterion. You could think of the information criteria as stylized mappings from a Bayesian posterior mass function in the model space to a real number. This is consistent with Cox's axioms.

If these were viewed as Bayesian tools, then they would create not a relative number but, instead, a probability statement that a particular model is the true model in nature.

Imagine you were considering the following possible models: $$M_1:z=\alpha+\epsilon$$ $$M_2:z=\beta_x{x}+\alpha+\epsilon$$ $$M_3:z=\beta_y{y}+\alpha+\epsilon$$ $$M_4:z=\beta_x{x}+\beta_y{y}+\alpha+\epsilon$$

In Bayesian model selection, each model, $M_x\in\mathcal{M}$, would be considered a parameter in the model space. A prior probability would be assigned to it.

The origin of your statements come from two things. First, in some circumstances, the AIC outperforms the BIC if the true model is not in the proposed model space, such as if there is an omitted variable in the above set. Conversely, in certain circumstances, the BIC outperforms the AIC when the true model is in the set of tested models.

Unfortunately, the discussion of this has a large and somewhat contradictory literature. It is way too large to summarize here. There are whole books on it. The focus on the AIC with a high dimensional model has to do with the penalty terms. The BIC penalizes complexity more than the AIC.

The prior probability for each model under the BIC is 1/4. Specifically, $\Pr(M_x)=\frac{1}{4}$.

If $n$ is large, then the prior probability of a model being the true model under the AIC specification is $$\Pr(M_x)=\frac{\exp(\frac{1}{2}k_i\log(n)-k_i)}{\sum_i^I\exp(\frac{1}{2}k_i\log(n)-k_i)}.$$

As the sample size becomes large and if the complexity stays low, as above, then the AIC and the BIC imply the approximately the same priors and they are proportionate to each other.

If you have not used Bayesian methods before then the formula is $$\Pr(M_x;\theta_x|D)\propto\mathcal{L}(D|M_x;\theta_x)\Pr(M_x;\theta_x)\forall\theta_x\in\Theta_x$$ where $D$ is data, $\theta_x$ are the set of parameters in Model X, $\Theta_x$ is the set of parameters in model x, and $M_x$ is the parameter denoting the model.

There is a process in Bayesian methods called marginalization where the effect of the parameters in a model are removed leaving only model probabilities. Here, it would be $$\Pr(M_x|D)=\int_{\theta_x\in\Theta}\Pr(M_x;\theta_x)\mathrm{d}\theta_x.$$

Under that specification $$\Pr(M_x|D)\to\Delta_x^{BIC}$$ where the prior probability is identical under all specifications and $$\Pr(M_x|D)\to\Delta_x^{AIC}$$ under the more complex prior shown above for the AIC.

Burnham, K. P., & Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretical Approach. 2d ed. New York: Springer-Verlag.

Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods and Research, 33(2), 261–304.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.