# Kullback-Leibler divergence from density f to density g

If $$f$$ and $$g$$ are density functions that are positive over the same region, then the Kullback-Leibler divergence from density $$f$$ to density $$g$$ is defined by:

$$KL(f,g) = E_f\left[\ln\left(\frac{f(x)}{g(x)}\right)\right] = \int\ln\left(\frac{f(x)}{g(x)}\right)f(x) dx$$

where the notation $$E_f[h(X)]$$ is used to indicate that $$X$$ has density function $$f$$.

1. Show that $$KL(f,f) = 0$$.

Answer: Using the given inequality, we obtain $$\ln(f/f) = \ln(1) = 0$$.

1. Show that $$KL(f,g) \geq 0$$. Hint: Use Jensen's Inequality.

Answer: Jensen's Inequality is: $$\phi(E[X]) \leq E[\phi(x)]$$

where $$\phi(x)$$ is a convex function. If we take $$\phi(x) = -\ln\left(\frac{g(x)}{f(x)}\right)$$, we can easily show that $$\phi(x)$$ is convex.

Substituting into Jensen's Inequality I obtain: $$\ln\left(\frac{g(E[X])}{f(E[X])}\right) \geq E\left[\ln\left(\frac{g(x)}{f(x)}\right)\right]$$

However, I am unsure how this can help us show $$KL(f,g)\geq 0$$.