Let $X_1$ be an observation from a normal random variable with mean zero and variance $\sigma^2$ and lets call the PDF $f(x)$.

I want to minimize the Kullback Liebler Information criterion between a PDF $g(x, \beta) $ of a zero mean normal random variable with variance $\beta \sigma^2$ and $f(x)$. The minimization is with respect to $\beta$.

The Kullback Liebler Information criterion is defined as t

$$I(f: g, \beta):= E [ \log(f(X_1)/g(X_1, \beta) )]$$


The motivation for doing this is that Akaike showed that the maximum likelihood estimator $\hat{\beta}$ of a model that assumes $g$ as the parametric distribution generating the observations is a natural estimator for the value $\beta^*$ that minimizes the Kullback Liebler information criterion.


I wanted to do this simple calculation because I expected $\beta^* = 1$ (in this way the two distributions would be equal so their "distance" is minimized). But performing the computations I obtain

$$ E \left[ \frac{1}{2} \log \beta X_1^2 - \frac{1}{2} \log \sigma^2 - \frac{\beta X_2^2 + \sigma^2}{2 \sigma^2 \beta} \right] $$

and it seems to me that the $\beta^* $ that minimizes this tends to minus infinity. Where is my mistake?

  • $\begingroup$ You didn't mention the form you want to assume for $g$ $\endgroup$ – user20160 Apr 20 at 13:14
  • $\begingroup$ @user20160 thanks, edited! $\endgroup$ – Monolite Apr 20 at 13:50

I am not so sure why there exist both "$X_1$" and "$X_2$" in your expression of KL information, but I think the following solution should be right.

According to the expression of KL information, we have $$I(f:g)=\int_x g(x)\log\left[\frac{f(x)}{g(x)}\right] dx=\int_xg(x)\left[\log(f(x))-\log(g(x))\right] dx.$$

Here "$\log$" refers to natural logarithms. Take the expression $f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{x^2}{2\sigma^2}}$ and $g(x)=\frac{1}{\sqrt{2\pi\beta\sigma^2}}e^{-\frac{x^2}{2\beta\sigma^2}}$ into the above expression, we have: $$I(f:g)=\int_xg(x)\left[-\frac{1}{2}\log(2\pi\sigma^2)-\frac{x^2}{2\sigma^2}+\frac{1}{2}\log(2\pi\beta\sigma^2)+\frac{x^2}{2\beta\sigma^2}\right] dx.$$

Due to the Gaussian distribution, we know that $\int_xg(x)x^2=\beta\sigma^2$ and $\int_xg(x)=1$. So the KL information can be expressed as: $$I(f:g)=\frac{1}{2}\log\left(\frac{2\pi\beta\sigma^2}{2\pi\sigma^2}\right)+\frac{\beta\sigma^2}{2\beta\sigma^2}-\frac{\beta\sigma^2}{2\sigma^2}=\frac{1}{2}\log\beta+\frac{1}{2}-\frac{\beta}{2}$$

Take derivation with respect to $\beta$ and set the result to 0, we can obtain that $$\frac{1}{2\beta}-\frac{1}{2}=0\rightarrow\beta=1$$

I think this is exactly what you want.

  • 1
    $\begingroup$ The $X_2$ is a typo. Thank you very much, very well put first answer! Welcome to the site. $\endgroup$ – Monolite Apr 20 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.