On a mistake computing the Kullback Liebler Information Criterion

Question

THE FRAMEWORK:

Let $X_1$ be an observation from a normal random variable with mean zero and variance $\sigma^2$ and lets call the PDF $f(x)$.

I want to minimize the Kullback Liebler Information criterion between a PDF $g(x, \beta) $ of a zero mean normal random variable with variance $\beta \sigma^2$ and $f(x)$. The minimization is with respect to $\beta$.

The Kullback Liebler Information criterion is defined as t

$$I(f: g, \beta):= E [ \log(f(X_1)/g(X_1, \beta) )]$$

THE MOTIVATION:

The motivation for doing this is that Akaike showed that the maximum likelihood estimator $\hat{\beta}$ of a model that assumes $g$ as the parametric distribution generating the observations is a natural estimator for the value $\beta^*$ that minimizes the Kullback Liebler information criterion.

THE PROBLEM:

I wanted to do this simple calculation because I expected $\beta^* = 1$ (in this way the two distributions would be equal so their "distance" is minimized). But performing the computations I obtain

$$ E \left[ \frac{1}{2} \log \beta X_1^2 - \frac{1}{2} \log \sigma^2 - \frac{\beta X_2^2 + \sigma^2}{2 \sigma^2 \beta} \right] $$

and it seems to me that the $\beta^* $ that minimizes this tends to minus infinity. Where is my mistake?

Answer 1

I am not so sure why there exist both "$X_1$" and "$X_2$" in your expression of KL information, but I think the following solution should be right.

According to the expression of KL information, we have $$I(f:g)=\int_x g(x)\log\left[\frac{f(x)}{g(x)}\right] dx=\int_xg(x)\left[\log(f(x))-\log(g(x))\right] dx.$$

Here "$\log$" refers to natural logarithms. Take the expression $f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{x^2}{2\sigma^2}}$ and $g(x)=\frac{1}{\sqrt{2\pi\beta\sigma^2}}e^{-\frac{x^2}{2\beta\sigma^2}}$ into the above expression, we have: $$I(f:g)=\int_xg(x)\left[-\frac{1}{2}\log(2\pi\sigma^2)-\frac{x^2}{2\sigma^2}+\frac{1}{2}\log(2\pi\beta\sigma^2)+\frac{x^2}{2\beta\sigma^2}\right] dx.$$

Due to the Gaussian distribution, we know that $\int_xg(x)x^2=\beta\sigma^2$ and $\int_xg(x)=1$. So the KL information can be expressed as: $$I(f:g)=\frac{1}{2}\log\left(\frac{2\pi\beta\sigma^2}{2\pi\sigma^2}\right)+\frac{\beta\sigma^2}{2\beta\sigma^2}-\frac{\beta\sigma^2}{2\sigma^2}=\frac{1}{2}\log\beta+\frac{1}{2}-\frac{\beta}{2}$$

Take derivation with respect to $\beta$ and set the result to 0, we can obtain that $$\frac{1}{2\beta}-\frac{1}{2}=0\rightarrow\beta=1$$

I think this is exactly what you want.