# How to measure similarity of bivariate probability distributions?

I have three different distributions of 2D data:

or

Now I like to know whether distribution two is more similar to distribution one (2 to 1) than distribution three is to distribution one (3 to 1)? What is the proper way to measure those similarities (and preferably express them in a single number)?

What I did / thought of so far:

1. As some kind of approximation for a similarity measure I used bounded bivariate kernel density estimation in a first step and then correlated the resulting PDFfs. However, this doesn't seem to be the most apprpriate way, since large regions of the PDFs are highly correlated (e.g., all of the low probability regions are ~0).

2. I have looked at using the Two-sample Kolmogorov–Smirnov test for 2D distributions; however, this only tells me whether the two distributions are significantly different and does not provide a likelihood measure that allows me to say that the data was better predicted by one distribution or the other.

3. Another method I thought of was fitting a curve to the data and simply measure the euclidean distance between the curves. However, here I don't know the proper way to fit a curve to 2D data. Besides, if I manage to fit a curve how do I determine corresponding points on the curves to measure the distance.

• Can you tell us why you want to compare these distributions/ what kind of conclusions you are trying to draw? This will surely influence which measure is the right one for you. – Lucas Feb 24 '15 at 9:22
• What is the 2nd set of plots? Are these the data in some transformed space? What are these data? Do you have any idea why the inconsistent straight lines of points jutting out from the side occur? – gung Feb 24 '15 at 17:43
• @ Lucas & gung:(1) The two different sets of plots are just two different independent sets of data. I just wanted to provide a second example. (2) A single plot within a set represents spatial events (e.g. 10 irregularly measured locations of a number of moving agents) under a specific condition (three different conditions in total). Now I want to know how similar the distributions of motion paths are under the different conditions. (e.g. is the distribution of location data under condition 2 more similar to condition 1 or to condition 3 based on a continuous number) – Lipton Ice Mar 6 '15 at 9:45

## 2 Answers

I suggest using the Jensen-Shannon divergence (JSD). For distributions $P$ and $Q$ it is given by $$D_\text{JS}[P, Q] = \frac{1}{2} D_\text{KL}[P \mid\mid M] + \frac{1}{2} D_\text{KL}[Q \mid\mid M],$$

where $M = \frac{1}{2}(P + Q)$ and $D_\text{KL}$ is the Kullback-Leibler divergence. Its advantages over other divergences are that it's symmetric, $\sqrt{D_\text{JS}[P, Q]}$ is a proper metric, and it's fairly intuitive because of its connection to mutual information*. It can also be generalized to more than two distributions if needed. For $P$ and $Q$ you can use the nonparametric estimates you already obtained.

*In a nutshell: Say I randomly pick $P$ or $Q$, both with 50% probability, draw one sample $x$ from it and give it to you. If you can tell whether $x$ came from $P$ or $Q$, there is a lot of information in $x$ about which distribution it belongs to. If you cannot tell, there is little information and the two distributions must be very similar. This is what the JSD measures.

You could subtract 2 and 3 each from 1, normalize the resulting distributions and eyeball them. I'll call the resulting distributions "difference distributions". Then you compare one to the other. The closer to a Dirac delta function at zero the better.

• I rephrased it as an answer. – jjack Feb 24 '15 at 17:22
• OK, thanks. It still seems a bit more like a comment than an answer, but I suppose it's answerish enough. I don't seem to be able to retract my delete vote (& I didn't downvote), though. – gung Feb 24 '15 at 17:47
• Given that these distributions evidently are scatterplots of sets of 2D points, could you please explain what it would mean to "subtract" one from another? – whuber Feb 24 '15 at 19:21
• Each frequency distribution gives the frequency of points in a bin around (x,y). This is a density of points in that bin of area ($\Delta{x}, \Delta{y}$). Subtracting distributions 2 and 3 each from distribution 1, one gets two "error distributions" (if we accept distribution 1 as our 'true' distribution for a moment). If we normalize each distribution to get relative frequencies at each bin, we can compare one resulting normalized error distribution to the other. The one that is closer to the origin at (0,0) with a sharper crest is the one that is more similar to distribution 1. – jjack Feb 24 '15 at 19:41
• You have to use the magnitude of the resulting difference distribution of course, since probabilities can't be negative. – jjack Feb 25 '15 at 20:05