# Kullback-Leibler divergence from density f to density g

If $$f$$ and $$g$$ are density functions that are positive over the same region, then the Kullback-Leibler divergence from density $$f$$ to density $$g$$ is defined by:

$$KL(f,g) = E_f\left[\ln\left(\frac{f(x)}{g(x)}\right)\right] = \int\ln\left(\frac{f(x)}{g(x)}\right)f(x) dx$$

where the notation $$E_f[h(X)]$$ is used to indicate that $$X$$ has density function $$f$$.

1. Show that $$KL(f,f) = 0$$.

Answer: Using the given inequality, we obtain $$\ln(f/f) = \ln(1) = 0$$.

1. Show that $$KL(f,g) \geq 0$$. Hint: Use Jensen's Inequality.

Answer: Jensen's Inequality is: $$\phi(E[X]) \leq E[\phi(x)]$$

where $$\phi(x)$$ is a convex function. If we take $$\phi(x) = -\ln\left(\frac{g(x)}{f(x)}\right)$$, we can easily show that $$\phi(x)$$ is convex.

Substituting into Jensen's Inequality I obtain: $$\ln\left(\frac{g(E[X])}{f(E[X])}\right) \geq E\left[\ln\left(\frac{g(x)}{f(x)}\right)\right]$$

However, I am unsure how this can help us show $$KL(f,g)\geq 0$$.

• Please add the self-study tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Please make these changes as just posting your homework & hoping someone will do it for you is grounds for closing. Commented Dec 10, 2021 at 2:34