# Relation between the AIC and the Kullback-Leibler Divergence

I am searching a formal derivation of the Akaike Information Criterion from the Kullback-Leibler Divergence. Can you show me one, or point me toward a book/article in which this is done?

Here I set the notation, but you can use your own if you wish

$$\begin{equation*} KL = E_{\theta_0}(l(\theta_0, Y)) - E_{\theta_0}(l(\theta, Y)) \quad \theta \in \Theta \end{equation*}$$ $$\begin{equation*} AIC = -2 l(\hat{\theta}, y) + 2p, \end{equation*}$$ where

• $$Y$$ is the random variable which generates the data;
• $$\theta_0$$ is the true value of the parameter;
• $$y$$ is a sample;
• $$\hat{\theta}$$ is the maximum likelihood estimate of $$\theta$$ computed using the sample $$y$$;
• $$p$$ is the dimension of the parameter space $$\Theta$$ where $$\theta$$ belongs.

What I know is that when $$\theta \neq \theta_0$$, we have that $$KL > 0$$ (result known as Kullback-Leibler Inequality). So a model is good when $$KL$$ is low. Then I know that the first summand of $$KL$$ is unknown because we do not know $$\theta$$, but when we want to compare some correctly specified nested models, since it is constant, we can drop it. So I guess that the $$AIC$$ is a sort of estimate of the second summand of $$KL$$.

• I am interested too in the answer, please, add some details if you find it. – ThePunisher Jan 28 at 14:08