Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler divergence are: $$ \mu_1 = \mathbb E_1[X] $$ and $$ \Sigma_1 = \mathbb E_1[(X − \mu)(X − \mu)^T], $$ where the notation $\mathbb E_1(\cdot)$ indicates the expectation is taken over the density $p_1(x)$?

For reference, the definition of Kullback–Leibler divergence is $$ D( p_1 \|\; p_2) = \int p_1 \log(p_1/p_2) \text{d}\lambda \> . $$

  • 2
    I have tried to edit your question to use $\LaTeX$ math notation. I've also tweaked the wording a little bit. Please make sure I have not inadvertently introduced errors. Cheers. :) – cardinal Feb 12 '12 at 20:40
  • One doubt: I'm not quite sure what your title for the question is supposed to be getting at. Can you explain briefly? – cardinal Feb 12 '12 at 20:43
  • All Im trying to say is that the difference between two distributions in the same space is the Kullback-Leibler divergence. – john Feb 12 '12 at 20:48
  • 1
    I guess that depends on what you mean by "difference". :) – cardinal Feb 12 '12 at 20:52

If you express the Kullback–Leibler divergence when $p_2$ is a normal pdf, $$ D(p_1||p_2) =\int p_1 \log p_1 \text{d}\lambda - \int p_1 \log p_2 \text{d}\lambda $$ $$ = \int p_1 \log p_1 \text{d}\lambda - \dfrac{1}{2} \int p_1 \left\{-(x-\mu)^T \Sigma^{-1} (x-\mu) - \log |\Sigma| -d \log 2\pi \right\} \text{d}\lambda $$ $$ = \int p_1 \log p_1 \text{d}\lambda + \dfrac{1}{2} \left\{ \log |\Sigma| + d \log 2\pi + \mathbb{E}_1 \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right] \right\} $$ Now $$ \mathbb{E}_1 \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right]= \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right]$$ $$\qquad\qquad\qquad + (\mathbb{E}_1[x]-\mu)^T \Sigma^{-1} (\mathbb{E}_1[x]-\mu) $$ so the minimum in $\mu$ is indeed reached for $\mu=\mathbb{E}_1[x]$.

Minimising $$ \log |\Sigma| + \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right] = $$ $$ \log |\Sigma| + \mathbb{E}_1 \left[ \text{trace} \left\{ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right\}\right] = \qquad\qquad\qquad $$ $$ \log |\Sigma| + \mathbb{E}_1 \left[ \text{trace} \left\{ \Sigma^{-1} (x-\mathbb{E}_1[x]) (x-\mathbb{E}_1[x] )^T \right\}\right] = $$ $$ \log |\Sigma| + \text{trace} \left\{ \Sigma^{-1} \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x]) (x-\mathbb{E}_1[x] )^T \right] \right\}= $$ $$ \qquad\qquad \log |\Sigma| + \text{trace} \left\{ \Sigma^{-1} \Sigma_1 \right\} $$ leads to a minimum in $\Sigma$ for $\Sigma=\Sigma_1$.

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.