If $f$ and $g$ are density functions that are positive over the same region, then the Kullback-Leibler divergence from density $f$ to density $g$ is defined by:

$$KL(f,g) = E_f\left[\ln\left(\frac{f(x)}{g(x)}\right)\right] = \int\ln\left(\frac{f(x)}{g(x)}\right)f(x) dx$$

where the notation $E_f[h(X)]$ is used to indicate that $X$ has density function $f$.

  1. Show that $KL(f,f) = 0$.

Answer: Using the given inequality, we obtain $\ln(f/f) = \ln(1) = 0$.

  1. Show that $KL(f,g) \geq 0$. Hint: Use Jensen's Inequality.

Answer: Jensen's Inequality is: $$\phi(E[X]) \leq E[\phi(x)]$$

where $\phi(x)$ is a convex function. If we take $\phi(x) = -\ln\left(\frac{g(x)}{f(x)}\right)$, we can easily show that $\phi(x)$ is convex.

Substituting into Jensen's Inequality I obtain: $$\ln\left(\frac{g(E[X])}{f(E[X])}\right) \geq E\left[\ln\left(\frac{g(x)}{f(x)}\right)\right]$$

However, I am unsure how this can help us show $KL(f,g)\geq 0$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.