1
$\begingroup$

How is the Akaike Information Criterion (AIC) related to Information theory ? I mean from the equation (below), it is not at all intuitive how information theory comes into picture. Also is AIC related to KL divergence? If yes, how ?

enter image description here

$\endgroup$
  • $\begingroup$ I am not home to write you an answer but the AIC is derived from information theory directly. It can be thought of as approximating the posterior at the maximum and the Bayesian posterior minimizes the kl divergence under stylized assumptions. $\endgroup$ – Dave Harris Oct 18 at 19:02
  • $\begingroup$ Sry, typed it backwards, minimizes. $\endgroup$ – Dave Harris Oct 18 at 19:02
  • $\begingroup$ Ignore last comment $\endgroup$ – Dave Harris Oct 18 at 19:03
  • $\begingroup$ Hmm.. Thanks for the reply Dave. Unfortunately, I am still not clear. My hunch is that the log likelihood mentioned in the equation (L hat) is the connecting piece to the information theory. But i just can't figure how. How come log likelihood be related to information loss? May be my question is cos of missing preceding steps in the derivation of this equation and just looking at the final equation $\endgroup$ – Ridhima Kumar Oct 18 at 19:10
1
$\begingroup$

The AIC is a creature of information theory. It began when Akaike realized that the maximum likelihood estimator is biased upward in $k$. It also happens that the bias is approximately equal to $k$ under a set of highly stylized circumstances. The AIC could be thought of as a constant minus the expected information. It is also approximately equal to the estimated relative K-L divergence.

What you are missing is the relationship between the likelihood principle and information. You can derive the K-L divergence directly from the Bayesian posterior predictive distribution and the true distribution. One is the transformation of the other. The posterior predictive distribution is the minimizing solution. Likewise, MLE methods follow the likelihood principle, but you may not be able to use the AIC as above.

For example, if the MLE is inconsistent as is the case in the Neyman-Scott problem. It doesn't mean that no information criterion exists for the problem, it just won't be the AIC.

While the likelihood principle is a highly debatable principle and depends heavily on how it is formulated, it might help to think about it here, noting that it could help you think about the link to information.

The likelihood principle is created from two axioms, the sufficiency principle and the conditionality principle.

The sufficiency principle links directly to information. Although this is not a formulation of it, it says if a sufficient statistic exists, then use it. If you had to make a decision that is impacted by a parameter $\theta$ and $\theta$ is strictly unknown, but you do have data, then you want to make your decision independent of $\theta$ because to do otherwise would require you to know $\theta$.

There are two ways to discuss sufficiency. One is as a statistic that is independent of $\theta$ but the other is to note that $\Pr(\theta|X)\propto\Pr(\theta|T(X))$. As the full data set is always sufficient, the parameter estimator $T$ has all the same information in it as $X$.

The second principle is subtle. If you have a collection of experiments about $\theta$ that could be performed and the decision as to which experiment to perform is chosen independent of $\theta$ then the experiments not chosen are irrelevant to inference.

That seems harmless but it isn't and the issue is contentious. Let us consider an example. You decide to perform inference on the average heights of fifth-grade boys by randomly selecting a school to measure its population of children. Let us assume there is believed to be no systematic reason the schools may vary from one another. The conditionality principle says the data that was not collected at the other schools is irrelevant to inference.

No data = no information.

Put together, the resulting principle is that all information about $\theta$ is contained in the likelihood function. Please note that I am not being rigorous. If someone really partisan came along, they might drop a mountain of criticisms. That isn't my point, nor am I arguing that the Likelihood principle is valid. I am arguing that you can see the linkage between the maximum likelihood estimator and information.

Now, from there, you can think of models as being a parameter and things such as slopes as nuisance parameters. Now it should be noted that I have left out a mountain of things, like why nobody uses the Mediocre Likelihood Estimator or the Trivially Small Likelihood estimator.

Consider the two distinct ways that exist in the mainstream to estimate the variance.

You could use the MLE of $\hat{\theta}_{MLE}=\frac{\sum{(X_i}-\bar{x})^2}{n}$ versus the MVUE of $\hat{\theta}_{MVUE}=\frac{\sum{(X_i-\bar{x})^2}}{n-1}$. The likelihood principle and the AIC imply the difference matters. The MVUE, in this case, is a mediocre likelihood estimator. As I said, I left a lot out in this discussion. Also, this post could be viewed as a bit partisan because it ignores the good reasons to use unbiased estimators, confidence intervals and things such as t-tests.

I didn't do derivations because this post would be a book chapter but they are readily found. The goal was an intuitive explanation of what I think you are missing.

$\endgroup$
  • $\begingroup$ Thanks Dave for such a clear and elaborate answer. It certainly cleared a lot of my doubts. $\endgroup$ – Ridhima Kumar Oct 19 at 4:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.