# Kullback–Leibler vs Kolmogorov-Smirnov distance

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions.

• Is there a typical situation where one should be used instead of the other?
• What is the rationale to do so?

The KL-divergence is typically used in information-theoretic settings, or even Bayesian settings, to measure the information change between distributions before and after applying some inference, for example. It's not a distance in the typical (metric) sense, because of lack of symmetry and triangle inequality, and so it's used in places where the directionality is meaningful.

The KS-distance is typically used in the context of a non-parametric test. In fact, I've rarely seen it used as a generic "distance between distributions", where the $\ell_1$ distance, the Jensen-Shannon distance, and other distances are more common.

• Another use of KL-divergence worth mentioning is in hypothesis testing. Assume $X_1, X_2, \ldots$ are iid from measures with density either $p_0$ or $p_1$. Let $T_n = n^{-1} \sum_{i=1}^n \log( p_1(X_i) / p_0(X_i) )$. By Neyman--Pearson, an optimal test rejects when $T_n$ is large. Now, under $p_0$, $T_n \to -D(p_0 \,\vert\vert\, p_1)$ in probability and under $p_1$, $T_n \to D(p_1 \,\vert\vert\, p_0)$. Since $D(\cdot \,\vert\vert\, \cdot)$ is nonnegative, the implication is that using the rule $T_n > 0$ to reject $p_0$ is asymptotically perfect. Apr 9, 2011 at 2:06
• Indeed. that's an excellent example. And in fact most general versions of the Chernoff-Hoeffding tail bounds use the KL-divergence. Apr 10, 2011 at 2:07
• @cardinal Is that an alternative to KS testing distribution equality?
– Dave
Jun 6, 2020 at 19:37

Another way of stating the same thing as the previous answer in more layman terms:

KL Divergence - Actually provides a measure of how big of a difference are two distributions from each other. As mentioned by the previous answer, this measure isnt an appropriate distance metric since its not symmetrical. I.e. distance between distribution A and B is a different value from the distance between distribution B and A.

Kolmogorov-Smirnov Test - This is an evaluation metric that looks at greatest separation between the cumulative distribution of a test distribution relative to a reference distribution. In addition, you can use this metric just like a z-score against the Kolmogorov distribution to perform a hypothesis test as to whether the test distribution is the same distribution as the reference. This metric can be used as a distance function as it is symmetric. I.e. greatest separation between CDF of A vs CDF of B is the same as greatest separation between CDF of B vs CDF of A.

KL divergence upper bounds Kolmogrov Distance and Total variation, meaning that if two distributions $$\mathcal{D}_1, \mathcal{D}_2$$ have a small KL divergence, then it follows that $$\mathcal{D}_1, \mathcal{D}_2$$ have a small total variation and subsequently a small Kolmogrov distance (in that order).