# What constitutes a large KL divergence?

I have 2 gamma distributions

$X_1 \sim Ga(13,1) \\ X_2 \sim Ga(3,1)$

Where we are defining our gamma distribution probability density function for a random variable $X \sim Ga(a,b)$ to be

$f(X) = \frac{b^a}{\Gamma(a)} x^{a-1} e^{-bx}$

ie $b$ is the rate.

Calculating the KL divergence for these 2 distributions gives me a value of around 10. I was just wondering would this be considered a large KL divergence? Is there some rule of thumb where you would consider this to be large therefore the 2 distributions to be different.

• Comments are not for extended discussion; this conversation has been moved to chat. – gung - Reinstate Monica Apr 2 '18 at 19:11
• Please continue the process of clarifying the question in the new chat room. The edits will bump this into the reopen queue. From there the community can reopen it as they deem fit. – gung - Reinstate Monica Apr 2 '18 at 19:14
• This KL divergence should be close to $0$, not $10$. I believe you have mixed up the shape and scale parameters in your interpretation of the referenced formulas. When you look at the formula in the right way this becomes fairly obvious, because $\Gamma$ and its logarithmic derivative $\psi$ must be applied to shape parameters $a$ and $c,$ not to the scale parameters. When I apply the formula at stats.stackexchange.com/a/11668/919 I obtain a KL divergence close to $0.6$, not $10.$ – whuber Apr 2 '18 at 20:34
• @whuber Yeah that's exactly what I did. I think I might delete this question. I just want to know the answer to the final part about the "rule of thumb" – user162934 Apr 2 '18 at 20:37
• That's an interesting question that I think has not been discussed on this site before. To avoid confusion, why not eliminate the details and just start out by stating you have obtained a KL divergence of $10$ (or so) between two Gamma distributions, and then proceed to ask how to interpret that? If you would like an example of such a pair of Gammas, use Gamma$(13,1)$ and Gamma$(3,1)$. – whuber Apr 2 '18 at 20:55

As commented by W. Huber, this is a fairly interesting question, even though I doubt there is a clear absolute answer. To quote a few generic references,

"...the K-L divergence represents the number of extra bits necessary to code a source whose symbols were drawn from the distribution P, given that the coder was designed for a source whose symbols were drawn from Q." Quora

and

"...it is the amount of information lost when Q is used to approximate P." Wikipedia

and

"The Kullback–Leibler divergence can also be interpreted as the expected discrimination information for $H_1$ over $H_0$: the mean information per sample for discriminating in favor of a hypothesis $H_1$ against a hypothesis $H_0$, when hypothesis $H_1$ is true." Wikipedia

But coding is a fairly specialised notion (in my opinion) while information is pretty vague (one could argue it is actually defined by the Kulback-Leibler distance). And there is no absolute scale since the distance most often ranges from 0 to $\infty$ (contrary to what the Wikipedia page may suggest in its first paragraph). Thus the scaling of calibration of a Kullback-Leibler distance will depend on the problem at hand and the reason why one measures such a distance.

An illustration of this calibration issue is provided in the following graph

which compares histograms of log-Kullback-Leibler distances between two Gamma distributions when

1. two datasets $x$ and $y$ of size $n$ are generated from a Gamma ${\cal G}(a,1)$
2. both parameters of the Gamma distribution are estimated from the samples by a method of moments
3. the Kullback-Leibler distance between the estimated Gammas is derived

Here is the core of the R code (using W. Huber's KL.gamma) in case this is unclear:

n=15
T=1e3
a=.3
diz=rep(0,T)
for (t in 1:T){
x=rgamma(n,17,1)
a=mean(x);b=var(x);a=a^2/b;b=sqrt(a/b)
y=rgamma(n,17,1)
c=mean(y);d=var(y);c=c^2/d;d=sqrt(c/d)
diz[t]=KL.gamma(a,b,c,d)}

The interpretation of this small experiment is that, when considering a sample of size $n=15$, a Kullback-Leibler divergence around $1$ is not significant in the sense that the same "true" parameters produce samples that lead to estimated distributions at a distance of around $1$. When moving to a sample of size $n=150$, this becomes a highly significant distance. (Note that this experiment is only trying to make a point of the lack of absolute "large" or "small" Kullback-Leibler divergence, not to turn this assessment of scale into a test or something like that!)

• There is clearly a relation between likelihood ratio test and KL divergence. Even though an absolute KL figure may not indicate much, with respect to a base model it will make sense. What do you think? – Cagdas Ozgenc Apr 25 '18 at 11:55
• @CagdasOzgenc: certainly, since it is an expected density ratio. However, the OP does not mention any data or even data size, which makes the 10 a unit-free quantity... – Xi'an Apr 25 '18 at 12:20