# Measuring the “distance” between two multivariate distributions

I'm looking for some good terminology to describe what I'm trying to do, to make it easier to look for resources.

So, say I have two clusters of points A and B, each associated to two values, X and Y, and I want to measure the "distance" between A and B - i.e. how likely is it that they were sampled from the same distribution (I can assume that the distributions are normal). For example, if X and Y are correlated in A but not in B, the distributions are different.

Intuitively, I would get the covariance matrix of A, and then look at how likely each point in B is to fit in there, and vice-versa (probably using someting like Mahalanobis distance).

But that is a bit "ad-hoc", and there is probably a more rigorous way of describing this (of course, in practice I have more than two datasets with more than two variables - I'm trying to identify which of my datasets are outliers).

Thanks!

• Dunno why, but a Mantel test flashed in front of my eyes when I read your post. – Roman Luštrik Nov 6 '10 at 15:43

There is also the Kullback-Leibler divergence, which is related to the Hellinger Distance you mention above.

• can one calculate the Kullback-Leibler divergence of points without making an assumption of the underlying probability density the points came from ? – Andre Holzner Nov 6 '10 at 16:22

Hmm, the Bhattacharyya distance seems to be what I'm looking for, though the Hellinger distance works too.

• you mention Bhattacharyya and Helling then accept an answer speaking about KL... At the end what was your choice and why? – Simon C. Jan 23 '19 at 16:20
• I believe it was KL divergence, but ... that was in 2010 and my memory is far from perfect. – Emile Jan 23 '19 at 17:21
• ahah yes I guessed that, but thank you anyway! – Simon C. Jan 23 '19 at 17:40

Heuristic

• Minkowski-form
• Weighted-Mean-Variance (WMV)

Nonparametric test statistics

• 2 (Chi Square)
• Kolmogorov-Smirnov (KS)
• Cramer/von Mises (CvM)

Information-theory divergences

• Kullback-Liebler (KL)
• Jensen–Shannon divergence (metric)
• Jeffrey-divergence (numerically stable and symmetric)

Ground distance measures

• Histogram intersection
• Earth Movers Distance (EMD)

The most complete survey is provided in Statistical Inference Based on Divergence Measures by Leandro Pardo, Complutense University, Chapman Hall 2006.

Few more measures of "Statistical Difference"

• Permutation test (by Fisher)
• Central Limit Theorem & Slutsky’s theorem
• Mann-Whitney-Wilcoxin test
• Anderson–Darling test
• Shapiro–Wilk test
• Hosmer–Lemeshow test
• Kuiper's test
• kernelized Stein discrepancy
• Jaccard similarity
• Also, hierarchical clustering deals with similarity measures between groups. The most popular measures of group similarity are perhaps the single linkage, complete linkage, and average linkage.