Kullback-Leibler divergence lower bound

Question

Are there any (nontrivial) lower bounds on the KL-divergence between two densities? Informally, I am trying to study problems where $f$ is some target density, and I want to show that if $g$ is chosen "poorly", then $KL(f\Vert g)$ must be large. Examples of "poor" behaviour could include different means, moments, etc.

Harder question: Any lower bounds on the KL-divergence between two mixture models (e.g. mixture of gaussians)? For example, a lower bound in terms of the component mixtures.

The only bound I have found is equation (19) in this paper, which unfortunately doesn't seem to help.

Answer 1

$\mbox{KL}(f||g)\geq 0$. But seriously, this is actually a really hard problem. Relevant to this topic is the area of Large Deviations Theory, specifically Rate Functions. You'll find a compendum of bounds here for example:

https://en.wikipedia.org/wiki/Inequalities_in_information_theory#Lower_bounds_for_the_Kullback.E2.80.93Leibler_divergence

with Kullback's Inequality being one such bound. The issue is always figuring out how the heck to calculate the rate function.

A slightly non-trivial bound is Pinsker's inequality, which says that total variation can be used to bound KL from belowi, but this is hardly ever tight.

Answer 2

I derived a lower bound which only depends on moments (e.g. mean and variance).

Even if the true distribution is unkown, we can calculate the lower bound (approximation) of the KL-divergence using only the expected value and the variance of a function we choose.

Please see Theorem 1 in the following paper.

https://arxiv.org/abs/1907.00288

URL of a sample code for this paper is

https://github.com/nissy220/KL_divergence

Please confirm the results.

In the left graph, the red line is the result of our lower bound and the blue line is the KL-divergence for the normal distribution.

The right graph displays the ratio.