# On a mistake computing the Kullback Liebler Information Criterion

THE FRAMEWORK:

Let $$X_1$$ be an observation from a normal random variable with mean zero and variance $$\sigma^2$$ and lets call the PDF $$f(x)$$.

I want to minimize the Kullback Liebler Information criterion between a PDF $$g(x, \beta)$$ of a zero mean normal random variable with variance $$\beta \sigma^2$$ and $$f(x)$$. The minimization is with respect to $$\beta$$.

The Kullback Liebler Information criterion is defined as t

$$I(f: g, \beta):= E [ \log(f(X_1)/g(X_1, \beta) )]$$

THE MOTIVATION:

The motivation for doing this is that Akaike showed that the maximum likelihood estimator $$\hat{\beta}$$ of a model that assumes $$g$$ as the parametric distribution generating the observations is a natural estimator for the value $$\beta^*$$ that minimizes the Kullback Liebler information criterion.

THE PROBLEM:

I wanted to do this simple calculation because I expected $$\beta^* = 1$$ (in this way the two distributions would be equal so their "distance" is minimized). But performing the computations I obtain

$$E \left[ \frac{1}{2} \log \beta X_1^2 - \frac{1}{2} \log \sigma^2 - \frac{\beta X_2^2 + \sigma^2}{2 \sigma^2 \beta} \right]$$

and it seems to me that the $$\beta^*$$ that minimizes this tends to minus infinity. Where is my mistake?

• You didn't mention the form you want to assume for $g$ – user20160 Apr 20 at 13:14
• @user20160 thanks, edited! – Monolite Apr 20 at 13:50

I am not so sure why there exist both "$$X_1$$" and "$$X_2$$" in your expression of KL information, but I think the following solution should be right.

According to the expression of KL information, we have $$I(f:g)=\int_x g(x)\log\left[\frac{f(x)}{g(x)}\right] dx=\int_xg(x)\left[\log(f(x))-\log(g(x))\right] dx.$$

Here "$$\log$$" refers to natural logarithms. Take the expression $$f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{x^2}{2\sigma^2}}$$ and $$g(x)=\frac{1}{\sqrt{2\pi\beta\sigma^2}}e^{-\frac{x^2}{2\beta\sigma^2}}$$ into the above expression, we have: $$I(f:g)=\int_xg(x)\left[-\frac{1}{2}\log(2\pi\sigma^2)-\frac{x^2}{2\sigma^2}+\frac{1}{2}\log(2\pi\beta\sigma^2)+\frac{x^2}{2\beta\sigma^2}\right] dx.$$

Due to the Gaussian distribution, we know that $$\int_xg(x)x^2=\beta\sigma^2$$ and $$\int_xg(x)=1$$. So the KL information can be expressed as: $$I(f:g)=\frac{1}{2}\log\left(\frac{2\pi\beta\sigma^2}{2\pi\sigma^2}\right)+\frac{\beta\sigma^2}{2\beta\sigma^2}-\frac{\beta\sigma^2}{2\sigma^2}=\frac{1}{2}\log\beta+\frac{1}{2}-\frac{\beta}{2}$$

Take derivation with respect to $$\beta$$ and set the result to 0, we can obtain that $$\frac{1}{2\beta}-\frac{1}{2}=0\rightarrow\beta=1$$

I think this is exactly what you want.

• The $X_2$ is a typo. Thank you very much, very well put first answer! Welcome to the site. – Monolite Apr 20 at 16:10