# How to measure the statistical “distance” between two frequency distributions?

I am undertaking a data analysis project which involves investigating website usage times over the course of the year. What I would like to do is compare how "consistent" the usage patterns are, say, how close they are to a pattern which involves using it for 1 hour once per week, or one which involves using it for 10 minutes a time, 6 times per week. I am aware of several things which can be calculated:

• Shannon entropy: measures how much the "certainty" in the outcome differs, i.e. how much a probability distribution differs from one that is uniform;
• Kullback-Liebler divergence: measures how much one probability distribution differs from another
• Jensen-Shannon divergence: similar to the KL-divergence, but more useful as it returns finite values
• Smirnov-Kolmogorov test: a test to determine whether two cumulative distribution functions for continuous random variables come from the same sample.
• Chi-squared test: a goodness-of-fit test to decide how well a frequency distribution differs from an expected frequency distribution.

What I would like to do is compare how much the actual usage durations (blue) differ from ideal usage times (orange) in distribution. These distributions are discrete, and the versions below are normalised to become probability distributions. The horizontal axis represents the amount of time (in minutes) a user has spent on the website; this has been recorded for each day of the year; if the user has not gone on the website at all then this counts as a zero duration but these have been removed from the frequency distribution. On the right is the cumulative distribution function.

My only problem is, even though I can get the JS-divergence to return a finite value, when I look at different users and compare their usage distributions to the ideal one, I get values that are mostly identical (which is therefore not a good indicator of how much they differ). Also, quite a bit of information is lost when normalising to probability distributions rather than frequency distributions (say a student uses the platform 50 times, then the blue distribution should be vertically scaled so that the total of the lengths of the bars equals 50, and the orange bar should have a height of 50 rather than 1). Part of what we mean by "consistency" is whether how often a user goes on the website affects how much they get out of it; if the number of times they visit the website is lost then comparing probability distributions is a bit dubious; even if the probability distribution of a user's duration is close to the "ideal" usage, that user may only have used the platform for 1 week during the year, which arguably is not very consistent.

Are there any well-established techniques for comparing two frequency distributions and calculating some sort of metric which characterises how similar (or dissimilar) they are?

• You might want to start by asking yourself what your loss function is (that is, in what way is usage pattern differing from the ideal bad, and how does the amount of badness depend on what sort of divergence there is), and designing your metric around that. – Acccumulation Sep 5 '19 at 17:37

You may be interested in the Earth mover's distance, also known as the Wasserstein metric. It is implemented in R (look at the emdist package) and in Python. We also have a number of threads on it.

The EMD works for both continuous and discrete distributions. The emdist package for R works on discrete distributions.

The advantage over something like a $$\chi^2$$ statistic is that the EMD yields interpretable results. Picture your distribution as mounds of earth, then the EMD tells you how much earth you would need to transport how far to turn one distribution into the other.

Put another way: two distributions (1,0,0) and (0,1,0) should be "more similar" than (1,0,0) and (0,0,1). The EMD will recognize this and assign a smaller distance to the first pair than to the second. The $$\chi^2$$ statistic will assign the same distance to both pairs, because it has no notion of an ordering in the distribution entries.

• Why that particular distance? It seems to be designed for any continuous distribution. OP has a frequency distribution, so why not a more "discrete" distance such as Chi-square? – user2974951 Sep 5 '19 at 8:47
• @user2974951: fair enough. See my edit. – Stephan Kolassa Sep 5 '19 at 8:56
• if distributions are 1D, as suggested in the question, then running and EMD solver is overkill. What needs to be done in that case is just compute a $L_p$ metric between the empirical quantile functions of both densities (essentially the inverses of the CDFs plotted in the quesion). p.31 in arxiv.org/abs/1803.00567 or if you need a more detailed account chapter 2 of math.u-psud.fr/~filippo/OTAM-cvgmt.pdf – Marco Cuturi Sep 10 '19 at 20:42
• @MarcoCuturi: an $L^p$ distance is certainly another possibility. However, it will again assign the same distance between (1,0,0) and (0,1,0) as between (1,0,0) and (0,0,1), which is a bit unintuitive. Whether the added intuitive niceness of the EMD is worth the additional complexity is something the OP can consider. – Stephan Kolassa Sep 11 '19 at 8:57

If you randomly sample an individual from each of the two distributions, you can calculate a difference between them. If you repeat this (with replacement) a number of times, you can generate a distribution of differences that contains all the information you are after. You can plot this distribution and characterize it with any summary statistics you desire - means, medians, etc.

• Is there a name for such a procedure? – user2974951 Sep 5 '19 at 8:55
• I wonder how one could account for the baseline fact that the distribution of differences for an arbitrary distribution and itself will be different for different arbitrary distributions; think U(0,1) vs. itself compared to N(0,1) vs. itself. Hence, the distribution of differences you would get comparing two different distributions would be hard to evaluate in absence of a unique baseline. The problem disappears if the observations are paired, then the baseline would be a unit mass at zero. – Richard Hardy Sep 5 '19 at 10:44
• @user2974951 I'm sure there is, as it's pretty simple and clearly related to bootstrapping. But I don't know what to call it precisely. – mkt - Reinstate Monica Sep 5 '19 at 11:02
• @mkt, thanks for your clarification. Without intending to argue just for the sake it, I still think that without a unique baseline we do not quite have a ruler. But I will leave it at that. There is something nice about your idea anyway. – Richard Hardy Sep 5 '19 at 15:18
• @RichardHardy I appreciate the exchange here, and you might well be right. I'll have to think about this further. – mkt - Reinstate Monica Sep 5 '19 at 15:42

One of the metric is Hellinger distance between two distributions which are characterised by means and standard deviations. The application can be found in the following article.

https://www.sciencedirect.com/science/article/pii/S1568494615005104

• Thanks for that. I have seen that there is a whole family of divergences (f-divergences) which do what I want, but a quick flick through the literature doesn't seem to indicate which is best when... do you know of any good literature on this? – omegaSQU4RED Sep 11 '19 at 13:17