When KL Divergence and KS test will show inconsistent results?

Question

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are different and should be used in different scenarios. But they are similar in many ways and given two distributions, we could calculate their KL divergence in terms of bits and p-value under KS test (and there are also other metrics like Jensen–Shannon divergence and many other hypothesis testing methods. But Let's just talk about KL divergence and KS test here.)

My question is: Under what circumstances, that KS test will provide a very small p-value but KL divergence will give us a very small distance? What is the intuition behind it? It would be better if there could be any concrete examples.

Answer 1

Set aside Kullback-Leibler divergence for a moment and consider the following: it's perfectly possible for the Kolmogorov-Smirnov p-value to be small and for the corresponding Kolomogorov-Smirnov distance to be small.

Specifically, that can easily happen with large sample sizes, where even small differences are still larger than we'd expect to see from random variation.

The same will naturally tend to happen when considering some other suitable measure of divergence and comparing it to the Kolmogorov-Smirnov p-value - it will quite naturally occur at large sample sizes.

[If you don't wish to confound the distinction between Kolmogorov-Smirnov distance and p-value with the difference in what the two things are looking at, it might be better to explore the differences in the two measures ($D_{KS}$ and $D_{KL}$) directly, but that's not what is being asked here.]