Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler divergence are: $$ \mu_1 = \mathbb E_1[X] $$ and $$ \Sigma_1 = \mathbb E_1[(X − \mu)(X − \mu)^T], $$ where the notation $\mathbb E_1(\cdot)$ indicates the expectation is taken over the density $p_1(x)$?

For reference, the definition of Kullback–Leibler divergence is $$ D( p_1 \|\; p_2) = \int p_1 \log(p_1/p_2) \text{d}\lambda \> . $$

  • 3
    $\begingroup$ I have tried to edit your question to use $\LaTeX$ math notation. I've also tweaked the wording a little bit. Please make sure I have not inadvertently introduced errors. Cheers. :) $\endgroup$
    – cardinal
    Feb 12, 2012 at 20:40
  • $\begingroup$ One doubt: I'm not quite sure what your title for the question is supposed to be getting at. Can you explain briefly? $\endgroup$
    – cardinal
    Feb 12, 2012 at 20:43
  • $\begingroup$ All Im trying to say is that the difference between two distributions in the same space is the Kullback-Leibler divergence. $\endgroup$
    – john
    Feb 12, 2012 at 20:48
  • 1
    $\begingroup$ I guess that depends on what you mean by "difference". :) $\endgroup$
    – cardinal
    Feb 12, 2012 at 20:52
  • $\begingroup$ I guess the divergence is not a metric on the space of distributions, right? $\endgroup$
    – MSIS
    Nov 13, 2020 at 4:29

1 Answer 1


If you express the Kullback–Leibler divergence when $p_2$ is a normal pdf on $\mathbb R^d$, \begin{align} D&(p_1||p_2) =\int_{\mathbb R^d} p_1 \log p_1 \text{d}\lambda - \int_{\mathbb R^d} p_1 \log p_2 \text{d}\lambda\\ &= \int_{\mathbb R^d} p_1 \log p_1 \text{d}\lambda - \dfrac{1}{2} \int_{\mathbb R^d} p_1 \left\{-(x-\mu)^T \Sigma^{-1} (x-\mu) - \log |\Sigma| -d \log 2\pi \right\} \text{d}\lambda \\ &= \int_{\mathbb R^d} p_1 \log p_1 \text{d}\lambda + \dfrac{1}{2} \left\{ \log |\Sigma| + d \log 2\pi + \mathbb{E}_1 \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right] \right\} \end{align} Now $$ \mathbb{E}_1 \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right]= \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right]$$ $$\qquad\qquad\qquad + (\mathbb{E}_1[x]-\mu)^T \Sigma^{-1} (\mathbb{E}_1[x]-\mu) $$ so the minimum in $\mu$ is indeed reached for $\mu=\mathbb{E}_1[x]$.

Minimising $$ \log |\Sigma| + \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right] = $$ $$ \log |\Sigma| + \mathbb{E}_1 \left[ \text{trace} \left\{ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right\}\right] = \qquad\qquad\qquad $$ $$ \log |\Sigma| + \mathbb{E}_1 \left[ \text{trace} \left\{ \Sigma^{-1} (x-\mathbb{E}_1[x]) (x-\mathbb{E}_1[x] )^T \right\}\right] = $$ $$ \log |\Sigma| + \text{trace} \left\{ \Sigma^{-1} \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x]) (x-\mathbb{E}_1[x] )^T \right] \right\}= $$ $$ \qquad\qquad \log |\Sigma| + \text{trace} \left\{ \Sigma^{-1} \Sigma_1 \right\} $$ leads to a minimum in $\Sigma$ for $\Sigma=\Sigma_1$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.