What constitutes a large KL divergence?

Question

I have 2 gamma distributions

$X_1 \sim Ga(13,1) \\ X_2 \sim Ga(3,1)$

Where we are defining our gamma distribution probability density function for a random variable $X \sim Ga(a,b)$ to be

$f(X) = \frac{b^a}{\Gamma(a)} x^{a-1} e^{-bx}$

ie $b$ is the rate.

Calculating the KL divergence for these 2 distributions gives me a value of around 10. I was just wondering would this be considered a large KL divergence? Is there some rule of thumb where you would consider this to be large therefore the 2 distributions to be different.

Answer 1

As commented by W. Huber, this is a fairly interesting question, even though I doubt there is a clear absolute answer. To quote a few generic references,

"...the K-L divergence represents the number of extra bits necessary to code a source whose symbols were drawn from the distribution P, given that the coder was designed for a source whose symbols were drawn from Q." Quora

and

"...it is the amount of information lost when Q is used to approximate P." Wikipedia

and

"The Kullback–Leibler divergence can also be interpreted as the expected discrimination information for $H_1$ over $H_0$: the mean information per sample for discriminating in favor of a hypothesis $H_1$ against a hypothesis $H_0$, when hypothesis $H_1$ is true." Wikipedia

But coding is a fairly specialised notion (in my opinion) while information is pretty vague (one could argue it is actually defined by the Kulback-Leibler distance). And there is no absolute scale since the distance most often ranges from 0 to $\infty$ (contrary to what the Wikipedia page may suggest in its first paragraph). Thus the scaling of calibration of a Kullback-Leibler distance will depend on the problem at hand and the reason why one measures such a distance.

An illustration of this calibration issue is provided in the following graph

which compares histograms of log-Kullback-Leibler distances between two Gamma distributions when

1. two datasets $x$ and $y$ of size $n$ are generated from a Gamma ${\cal G}(a,1)$
2. both parameters of the Gamma distribution are estimated from the samples by a method of moments
3. the Kullback-Leibler distance between the estimated Gammas is derived

Here is the core of the R code (using W. Huber's KL.gamma) in case this is unclear:

n=15
T=1e3
a=.3
diz=rep(0,T)
for (t in 1:T){
  x=rgamma(n,17,1)
  a=mean(x);b=var(x);a=a^2/b;b=sqrt(a/b)
  y=rgamma(n,17,1)
  c=mean(y);d=var(y);c=c^2/d;d=sqrt(c/d)
  diz[t]=KL.gamma(a,b,c,d)}

The interpretation of this small experiment is that, when considering a sample of size $n=15$, a Kullback-Leibler divergence around $1$ is not significant in the sense that the same "true" parameters produce samples that lead to estimated distributions at a distance of around $1$. When moving to a sample of size $n=150$, this becomes a highly significant distance. (Note that this experiment is only trying to make a point of the lack of absolute "large" or "small" Kullback-Leibler divergence, not to turn this assessment of scale into a test or something like that!)