I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are different and should be used in different scenarios. But they are similar in many ways and given two distributions, we could calculate their KL divergence in terms of bits and p-value under KS test (and there are also other metrics like Jensen–Shannon divergence and many other hypothesis testing methods. But Let's just talk about KL divergence and KS test here.)

My question is: Under what circumstances, that KS test will provide a very small p-value but KL divergence will give us a very small distance? What is the intuition behind it? It would be better if there could be any concrete examples.

  • 1
    $\begingroup$ For a comparison of Kolmogorov distance and KL divergence note that Kolmogorov distance compares values of cumulative distribution functions while KL divergence compares (log) ratio of densities. $\endgroup$ Commented Jun 27, 2018 at 8:27

1 Answer 1


Set aside Kullback-Leibler divergence for a moment and consider the following: it's perfectly possible for the Kolmogorov-Smirnov p-value to be small and for the corresponding Kolomogorov-Smirnov distance to be small.

Specifically, that can easily happen with large sample sizes, where even small differences are still larger than we'd expect to see from random variation.

The same will naturally tend to happen when considering some other suitable measure of divergence and comparing it to the Kolmogorov-Smirnov p-value - it will quite naturally occur at large sample sizes.

[If you don't wish to confound the distinction between Kolmogorov-Smirnov distance and p-value with the difference in what the two things are looking at, it might be better to explore the differences in the two measures ($D_{KS}$ and $D_{KL}$) directly, but that's not what is being asked here.]


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.