If $f$ and $g$ are density functions that are positive over the same region, then the Kullback-Leibler divergence from density $f$ to density $g$ is defined by:

$$KL(f,g) = E_f\left[\ln\left(\frac{f(x)}{g(x)}\right)\right] = \int\ln\left(\frac{f(x)}{g(x)}\right)f(x) dx$$

where the notation $E_f[h(X)]$ is used to indicate that $X$ has density function $f$.

  1. Show that $KL(f,f) = 0$.

Answer: Using the given inequality, we obtain $\ln(f/f) = \ln(1) = 0$.

  1. Show that $KL(f,g) \geq 0$. Hint: Use Jensen's Inequality.

Answer: Jensen's Inequality is: $$\phi(E[X]) \leq E[\phi(x)]$$

where $\phi(x)$ is a convex function. If we take $\phi(x) = -\ln\left(\frac{g(x)}{f(x)}\right)$, we can easily show that $\phi(x)$ is convex.

Substituting into Jensen's Inequality I obtain: $$\ln\left(\frac{g(E[X])}{f(E[X])}\right) \geq E\left[\ln\left(\frac{g(x)}{f(x)}\right)\right]$$

However, I am unsure how this can help us show $KL(f,g)\geq 0$.

  • $\begingroup$ Please add the self-study tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Please make these changes as just posting your homework & hoping someone will do it for you is grounds for closing. $\endgroup$ Commented Dec 10, 2021 at 2:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.