60
$\begingroup$

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions.

  • Is there a typical situation where one should be used instead of the other?
  • What is the rationale to do so?
Improve this question
$\endgroup$
1

4 Answers 4

Reset to default
40
$\begingroup$

The KL-divergence is typically used in information-theoretic settings, or even Bayesian settings, to measure the information change between distributions before and after applying some inference, for example. It's not a distance in the typical (metric) sense, because of lack of symmetry and triangle inequality, and so it's used in places where the directionality is meaningful.

The KS-distance is typically used in the context of a non-parametric test. In fact, I've rarely seen it used as a generic "distance between distributions", where the $\ell_1$ distance, the Jensen-Shannon distance, and other distances are more common.

Improve this answer
$\endgroup$
3
  • 10
    $\begingroup$ Another use of KL-divergence worth mentioning is in hypothesis testing. Assume $X_1, X_2, \ldots$ are iid from measures with density either $p_0$ or $p_1$. Let $T_n = n^{-1} \sum_{i=1}^n \log( p_1(X_i) / p_0(X_i) )$. By Neyman--Pearson, an optimal test rejects when $T_n$ is large. Now, under $p_0$, $T_n \to -D(p_0 \,\vert\vert\, p_1)$ in probability and under $p_1$, $T_n \to D(p_1 \,\vert\vert\, p_0)$. Since $D(\cdot \,\vert\vert\, \cdot)$ is nonnegative, the implication is that using the rule $T_n > 0$ to reject $p_0$ is asymptotically perfect. $\endgroup$
    – cardinal
    Commented Apr 9, 2011 at 2:06
  • 1
    $\begingroup$ Indeed. that's an excellent example. And in fact most general versions of the Chernoff-Hoeffding tail bounds use the KL-divergence. $\endgroup$
    – Suresh Venkatasubramanian
    Commented Apr 10, 2011 at 2:07
  • $\begingroup$ @cardinal Is that an alternative to KS testing distribution equality? $\endgroup$
    – Dave
    Commented Jun 6, 2020 at 19:37
8
$\begingroup$

Another way of stating the same thing as the previous answer in more layman terms:

KL Divergence - Actually provides a measure of how big of a difference are two distributions from each other. As mentioned by the previous answer, this measure isnt an appropriate distance metric since its not symmetrical. I.e. distance between distribution A and B is a different value from the distance between distribution B and A.

Kolmogorov-Smirnov Test - This is an evaluation metric that looks at greatest separation between the cumulative distribution of a test distribution relative to a reference distribution. In addition, you can use this metric just like a z-score against the Kolmogorov distribution to perform a hypothesis test as to whether the test distribution is the same distribution as the reference. This metric can be used as a distance function as it is symmetric. I.e. greatest separation between CDF of A vs CDF of B is the same as greatest separation between CDF of B vs CDF of A.

Improve this answer
$\endgroup$
3
$\begingroup$

KS test and KL divergence test both are used to find the difference between two distributions KS test is statistical-based and KL divergence is information theory-based But the one major diff between KL and KS test, and why KL is more popular in machine learning is because the formulation for KL divergence is differentiable. And for solving optimization problems in machine learning we need a function to be differentiable. In the context of machine learning, KL_dist(P||Q) is often called the information gain achieved if Q is used instead of P

links: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Improve this answer
$\endgroup$
3
$\begingroup$

KL divergence upper bounds Kolmogrov Distance and Total variation, meaning that if two distributions $\mathcal{D}_1, \mathcal{D}_2$ have a small KL divergence, then it follows that $\mathcal{D}_1, \mathcal{D}_2$ have a small total variation and subsequently a small Kolmogrov distance (in that order).

Also check out this paper for more information - On Choosing and Bounding Probability Metrics, by Gibbs and Su.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Welcome to CV. Please add the full reference of the paper in case your link dies in the future $\endgroup$
    – Antoine
    Commented Dec 30, 2020 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.