I was hoping someone knows whether the following statement is true or not.

Suppose $F$ is a given normal distribution and $G$ is a distribution that has a given mean $\mu$ and variance $\sigma^2$ (but can otherwise be anything). Then, the distribution G that has the minimum KL distance from F is a normal distribution (of course, $G$ would be a normal distribution mean $\mu$ and variance $\sigma^2$).

Is this true and if so, is there a reference? Really appreciate any assistance.


Let me share my attempt in the real-valued case for simplicity. Since $F$ has a density $f$ and we want to minimize the KL to $F$ it is enough to look for distributions $G$ with some density $g$. We want to find $g$ such that \begin{align} g \in \arg\min &\int g(x) \log\left(\frac{g(x)}{f(x)}\right) dx \qquad \text{subject to }\\ \int g(x) dx = 1, \qquad &\int x g(x) dx = \mu, \qquad\int x^2 g(x) dx = \sigma^2 \end{align} for some fixed $\mu, \sigma^2$. This is a standard variational problem with integral constraints, and we know that for $g$ to be an extremum it must solve the Euler-Lagrange equation \begin{equation} \frac{\partial}{\partial g} \left[ g(x)\log\left(\frac{g(x)}{f(x)}\right) + \lambda g(x) + \phi x g(x) + \theta x^2 g(x)\right] = 0 \end{equation} for some constants $\lambda, \theta, \phi$. Above display becomes \begin{equation} 1 + \log\left(\frac{g(x)}{f(x)}\right) + \lambda + \phi x + \theta x^2 = 0. \end{equation} From here it follows that $g(x) \propto f(x) \exp\left(-1-\lambda-\phi x - \theta x^2\right)$. In particular, g will be a Gaussian density.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.