20 votes
Accepted

Can KL-Divergence ever be greater than 1?

The Kullback-Leibler divergence is unbounded. Indeed, since there is no lower bound on the $q(i)$'s, there is no upper bound on the $p(i)/q(i)$'s. For instance, the Kullback-Leibler divergence ...
user avatar
  • 93.4k
17 votes

Hierarchical clustering with mixed type data - what distance/similarity to use?

If you have stumbled upon this question and are wondering what package to download for using Gower metric in R, the cluster package has a function named daisy(), ...
user avatar
  • 7,913
12 votes
Accepted

Intuition of the Bhattacharya Coefficient and the Bhattacharya distance?

The Bhattacharyya coefficient is $$ BC(h,g)= \int \sqrt{h(x) g(x)}\; dx $$ in the continuous case. There is a good wikipedia article https://en.wikipedia.org/wiki/Bhattacharyya_distance. How to ...
user avatar
11 votes
Accepted

If Manhattan distance always performs better on a dataset...what does it mean?

Also use the search terms l1 norm, l1 distance, absolute deviance etc all of which refer to the same thing as manhattan distance. The properties of the l1-norm (manhattan distance) can largely be ...
user avatar
10 votes

Coupling and Total variational distance

There are at least 2 ways to compute the total variation distance. The first is by using the definition of Total variation distance: $TV(\mu,\nu)=\sup_{ A\in \mathcal{F}}\left|\mu(A)-\nu(A)\right|,$ ...
user avatar
10 votes
Accepted

Jaccard similarity coefficient vs. Point-wise mutual information coefficient

These two are quite different. Still, let us try to "bring them to a common denominator", to see the difference. Both Jaccard and PMI could be extended to a continuous data case, but we'll observe the ...
user avatar
  • 52.9k
9 votes

Why does k-means clustering algorithm use only Euclidean distance metric?

I might be a little pedantic here, but K-means is the name given to a particular algorithm that assigns labels to data points such that within cluster variances are minimized, and it is not the name ...
user avatar
8 votes

Why does k-means clustering algorithm use only Euclidean distance metric?

Since this is apparently now a canonical question, and it hasn't been mentioned here yet: One natural extension of k-means to use distance metrics other than the standard Euclidean distance on $\...
user avatar
  • 22.4k
8 votes

What is the distribution of the Euclidean distance between two normally distributed random variables?

First define the bivariate distribution of the difference vector, $\mu_d = \mu_1 - \mu_2$, which will be simply $\Sigma_d = \Sigma_1 + \Sigma_2$; this follows from the multivariate uncertainty ...
user avatar
8 votes
Accepted

Jeffries Matusita distance for 14 variables

As Nick Cox points out in a comment to the question, the Jeffries-Matusita distance should be called the Jeffreys-Matusita distance due to its origin in the work of Harold Jeffreys. Whatever you ...
user avatar
  • 293k
8 votes
Accepted

Looking for a robust, distribution-free/nonparametric distance between multivariate samples

First of all, I advise you to take a look at the Encyclopedia of Distances by Michel and Elena Deza. From quickly browsing through the pdf (e.g. pp. 327-330), you can already see a multitude of ...
user avatar
  • 647
8 votes
Accepted

Similarity function with given properties

The function $$ f\colon [0,1]\times[0,1]\to[0,1], \quad(x,y)\mapsto \frac{1}{4}x+\frac{1}{4}y+\frac{3}{4}(x-y)^2 $$ does what you want. Plus, it's positive, symmetric and definite ($x\neq y$ implies ...
user avatar
8 votes
Accepted

Is the relative contrast theorem from Beyer et al. paper: "On the Surprising Behavior of Distance Metrics in High Dimensional Space" misleading?

No, the theorem is not misleading. It can certainly be applied incorrectly, but that's true for any theorem. Here's simple MATLAB script to demonstrate how it works: ...
user avatar
  • 56.8k
7 votes
Accepted

Is there an unbiased estimator of the Hellinger distance between two distributions?

No unbiased estimator either of $\mathfrak{H}$ or of $\mathfrak{H}^2$ exists for $f$ from any reasonably broad nonparametric class of distributions. We can show this with the beautifully simple ...
user avatar
  • 22.4k
7 votes
Accepted

Find K-nearest neighbour with custom distance metric

Yes it is. As stated by @Jeremie Clos, you can specify a custom metric. From the official documentation: ...
user avatar
  • 9,346
7 votes
Accepted

Similarity metrics for more than two vectors?

The cosine similarity between two column vectors $x_1$ and $x_2$ is simply the dot product between their unit vectors $$\mathrm{CosSim}[x_1,x_2]=\frac{x_1}{\|x_1\|}\bullet\frac{x_2}{\|x_2\|}$$ and ...
user avatar
  • 12.2k
6 votes

Choosing a clustering method

Choosing the right distance is not an elementary task. When we want to make a cluster analysis on a data set, different results could appear using different distances, so it's very important to be ...
user avatar
6 votes

Measuring distance between two empirical distributions

I think your question is essentially the same as Can I use Kolmogorov-Smirnov to compare two empirical distributions?, for which the Kolmogorov-Smirnov test is commonly used. The KS test statistic is ...
user avatar
6 votes

Kullback–Leibler vs Kolmogorov-Smirnov distance

Another way of stating the same thing as the previous answer in more layman terms: KL Divergence - Actually provides a measure of how big of a difference are two distributions from each other. As ...
user avatar
  • 209
6 votes
Accepted

advantage of euclidean distance for classification

That depends a lot on your use-case. If you're working in a continuous space where all dimensions are properly scaled and relevant, then Euclidean is going to be a great choice for distance function. ...
user avatar
  • 404
6 votes
Accepted

A similarity measure with binary data: does this one have a name?

Your measure seems to resolve to a distance defined by Simpson. See A Survey of Binary Similarity and Distance Measures page 44, equation 45.
user avatar
  • 2,052
6 votes
Accepted

Distance metric for source code

You can check the two links below: A comparison of code similarity analysers Measuring Code Similarity in Large-scaled Code Corpora At the third link below a similarity measure is proposed, which ...
user avatar
5 votes
Accepted

How to distance and to MDS-plot objects according their complex shape

This may be only a partial answer because I don't think the plot that you expect is really what is in the data, especially the "parallelity and continuity" of the intermediate signals. I will ...
user avatar
  • 8,920
5 votes

Why is Euclidean distance not a good metric in high dimensions?

As an analogy, imagine a circle centred at the origin. Points are distributed evenly. Suppose a randomly-selected point is at (x1, x2). The Euclidean distance from the origin is ((x1)^2 + (x2)^2)^0.5 ...
user avatar
5 votes

Ask for suggestions on clustering methods on a large dataset with mixed types of variables

Similarly to the previous answers, most of the following answer of mine is not specific to SAS, as I use R. However, there is one exception to that - please see below. It seems that there exist ...
user avatar
5 votes
Accepted

Online course about distance measures

Honestly I do not think that such narrowly focused course exists anywhere (online or offline), but there is an Encyclopedia of Distances book by Deza and Deza (2009, Springer) that you could check.
user avatar
  • 117k
5 votes
Accepted

square root missing in code?

The whole thread you linked, and the code you showed which was provided as an answer there, is in terms of Mahalanobis Distance squared, not Mahalanobis Distance. For certain purposes, it is ...
user avatar
5 votes
Accepted

Mahalanobis distance invariant against further elements/individuals?

It is only because you have a small sample size and thus a "poor" estimation of $S$, which may be greatly influenced by the new individuals. Try it out with $n < 10000$ and, as long as the new ...
user avatar
  • 414
5 votes
Accepted

Is there a version of the Mahalanobis distance for matrices?

No. There are metrics that try to build on a similar concept using Wishart distribution. I have seen papers in MRI imaging that use the metrics. See p.16 in this slide deck: https://earth.esa.int/c/...
user avatar
  • 56.8k
4 votes

Why does k-means clustering algorithm use only Euclidean distance metric?

I've read many interesting comments here, but let me add that Matlab's "personal" implementation of k-means supports 4 non-Euclidean distances [between data points and cluster centres]. The ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible