# Kullback-Leibler divergence

Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler divergence are: $$\mu_1 = \mathbb E_1[X]$$ and $$\Sigma_1 = \mathbb E_1[(X − \mu)(X − \mu)^T],$$ where the notation $\mathbb E_1(\cdot)$ indicates the expectation is taken over the density $p_1(x)$?

For reference, the definition of Kullback–Leibler divergence is $$D( p_1 \|\; p_2) = \int p_1 \log(p_1/p_2) \text{d}\lambda \> .$$

• I have tried to edit your question to use $\LaTeX$ math notation. I've also tweaked the wording a little bit. Please make sure I have not inadvertently introduced errors. Cheers. :) – cardinal Feb 12 '12 at 20:40
• One doubt: I'm not quite sure what your title for the question is supposed to be getting at. Can you explain briefly? – cardinal Feb 12 '12 at 20:43
• All Im trying to say is that the difference between two distributions in the same space is the Kullback-Leibler divergence. – john Feb 12 '12 at 20:48
• I guess that depends on what you mean by "difference". :) – cardinal Feb 12 '12 at 20:52

If you express the Kullback–Leibler divergence when $p_2$ is a normal pdf, $$D(p_1||p_2) =\int p_1 \log p_1 \text{d}\lambda - \int p_1 \log p_2 \text{d}\lambda$$ $$= \int p_1 \log p_1 \text{d}\lambda - \dfrac{1}{2} \int p_1 \left\{-(x-\mu)^T \Sigma^{-1} (x-\mu) - \log |\Sigma| -d \log 2\pi \right\} \text{d}\lambda$$ $$= \int p_1 \log p_1 \text{d}\lambda + \dfrac{1}{2} \left\{ \log |\Sigma| + d \log 2\pi + \mathbb{E}_1 \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right] \right\}$$ Now $$\mathbb{E}_1 \left[ (x-\mu)^T \Sigma^{-1} (x-\mu) \right]= \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right]$$ $$\qquad\qquad\qquad + (\mathbb{E}_1[x]-\mu)^T \Sigma^{-1} (\mathbb{E}_1[x]-\mu)$$ so the minimum in $\mu$ is indeed reached for $\mu=\mathbb{E}_1[x]$.
Minimising $$\log |\Sigma| + \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right] =$$ $$\log |\Sigma| + \mathbb{E}_1 \left[ \text{trace} \left\{ (x-\mathbb{E}_1[x] )^T \Sigma^{-1} (x-\mathbb{E}_1[x]) \right\}\right] = \qquad\qquad\qquad$$ $$\log |\Sigma| + \mathbb{E}_1 \left[ \text{trace} \left\{ \Sigma^{-1} (x-\mathbb{E}_1[x]) (x-\mathbb{E}_1[x] )^T \right\}\right] =$$ $$\log |\Sigma| + \text{trace} \left\{ \Sigma^{-1} \mathbb{E}_1 \left[ (x-\mathbb{E}_1[x]) (x-\mathbb{E}_1[x] )^T \right] \right\}=$$ $$\qquad\qquad \log |\Sigma| + \text{trace} \left\{ \Sigma^{-1} \Sigma_1 \right\}$$ leads to a minimum in $\Sigma$ for $\Sigma=\Sigma_1$.