I am searching a formal derivation of the Akaike Information Criterion from the Kullback-Leibler Divergence. Can you show me one, or point me toward a book/article in which this is done?

Here I set the notation, but you can use your own if you wish

\begin{equation*} KL = E_{\theta_0}(l(\theta_0, Y)) - E_{\theta_0}(l(\theta, Y)) \quad \theta \in \Theta \end{equation*} \begin{equation*} AIC = -2 l(\hat{\theta}, y) + 2p, \end{equation*} where

  • $Y$ is the random variable which generates the data;
  • $\theta_0$ is the true value of the parameter;
  • $y$ is a sample;
  • $\hat{\theta}$ is the maximum likelihood estimate of $\theta$ computed using the sample $y$;
  • $p$ is the dimension of the parameter space $\Theta$ where $\theta$ belongs.

What I know is that when $\theta \neq \theta_0$, we have that $KL > 0$ (result known as Kullback-Leibler Inequality). So a model is good when $KL$ is low. Then I know that the first summand of $KL$ is unknown because we do not know $\theta$, but when we want to compare some correctly specified nested models, since it is constant, we can drop it. So I guess that the $AIC$ is a sort of estimate of the second summand of $KL$.

  • $\begingroup$ I am interested too in the answer, please, add some details if you find it. $\endgroup$ – ThePunisher Jan 28 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.