20
votes
Accepted
Can KL-Divergence ever be greater than 1?
The Kullback-Leibler divergence is unbounded. Indeed, since there is no lower bound on the $q(i)$'s, there is no upper bound on the $p(i)/q(i)$'s. For instance, the Kullback-Leibler divergence ...
- 93.4k
17
votes
Hierarchical clustering with mixed type data - what distance/similarity to use?
If you have stumbled upon this question and are wondering what package to download for using Gower metric in R, the cluster package has a function named daisy(), ...
- 7,913
12
votes
Accepted
Intuition of the Bhattacharya Coefficient and the Bhattacharya distance?
The Bhattacharyya coefficient is
$$
BC(h,g)= \int \sqrt{h(x) g(x)}\; dx
$$
in the continuous case. There is a good wikipedia article https://en.wikipedia.org/wiki/Bhattacharyya_distance. How to ...
- 66.4k
11
votes
Accepted
If Manhattan distance always performs better on a dataset...what does it mean?
Also use the search terms l1 norm, l1 distance, absolute deviance etc all of which refer to the same thing as manhattan distance.
The properties of the l1-norm (manhattan distance) can largely be ...
- 2,167
10
votes
Coupling and Total variational distance
There are at least 2 ways to compute the total variation distance. The first is by using the definition of Total variation distance:
$TV(\mu,\nu)=\sup_{ A\in \mathcal{F}}\left|\mu(A)-\nu(A)\right|,$
...
- 3,969
10
votes
Accepted
Jaccard similarity coefficient vs. Point-wise mutual information coefficient
These two are quite different. Still, let us try to "bring them to a common denominator", to see the difference. Both Jaccard and PMI could be extended to a continuous data case, but we'll observe the ...
- 52.9k
9
votes
Why does k-means clustering algorithm use only Euclidean distance metric?
I might be a little pedantic here, but K-means is the name given to a particular algorithm that assigns labels to data points such that within cluster variances are minimized, and it is not the name ...
- 529
8
votes
Why does k-means clustering algorithm use only Euclidean distance metric?
Since this is apparently now a canonical question, and it hasn't been mentioned here yet:
One natural extension of k-means to use distance metrics other than the standard Euclidean distance on $\...
- 22.4k
8
votes
What is the distribution of the Euclidean distance between two normally distributed random variables?
First define the bivariate distribution of the difference vector, $\mu_d = \mu_1 - \mu_2$, which will be simply $\Sigma_d = \Sigma_1 + \Sigma_2$; this follows from the multivariate uncertainty ...
8
votes
Accepted
Jeffries Matusita distance for 14 variables
As Nick Cox points out in a comment to the question, the Jeffries-Matusita distance should be called the Jeffreys-Matusita distance due to its origin in the work of Harold Jeffreys.
Whatever you ...
- 293k
8
votes
Accepted
Looking for a robust, distribution-free/nonparametric distance between multivariate samples
First of all, I advise you to take a look at the Encyclopedia of Distances by Michel and Elena Deza. From quickly browsing through the pdf (e.g. pp. 327-330), you can already see a multitude of ...
- 647
8
votes
Accepted
Similarity function with given properties
The function
$$ f\colon [0,1]\times[0,1]\to[0,1], \quad(x,y)\mapsto \frac{1}{4}x+\frac{1}{4}y+\frac{3}{4}(x-y)^2 $$
does what you want. Plus, it's positive, symmetric and definite ($x\neq y$ implies ...
- 99.4k
8
votes
Accepted
Is the relative contrast theorem from Beyer et al. paper: "On the Surprising Behavior of Distance Metrics in High Dimensional Space" misleading?
No, the theorem is not misleading. It can certainly be applied incorrectly, but that's true for any theorem.
Here's simple MATLAB script to demonstrate how it works:
...
- 56.8k
7
votes
Accepted
Is there an unbiased estimator of the Hellinger distance between two distributions?
No unbiased estimator either of $\mathfrak{H}$ or of $\mathfrak{H}^2$ exists for $f$ from any reasonably broad nonparametric class of distributions.
We can show this with the beautifully simple ...
- 22.4k
7
votes
Accepted
Find K-nearest neighbour with custom distance metric
Yes it is. As stated by @Jeremie Clos, you can specify a custom metric. From the official documentation:
...
- 9,346
7
votes
Accepted
Similarity metrics for more than two vectors?
The cosine similarity between two column vectors $x_1$ and $x_2$ is simply the dot product between their unit vectors
$$\mathrm{CosSim}[x_1,x_2]=\frac{x_1}{\|x_1\|}\bullet\frac{x_2}{\|x_2\|}$$
and ...
- 12.2k
6
votes
Choosing a clustering method
Choosing the right distance is not an elementary task. When we want to make a cluster analysis on a data set, different results could appear using different distances, so it's very important to be ...
6
votes
Measuring distance between two empirical distributions
I think your question is essentially the same as Can I use Kolmogorov-Smirnov to compare two empirical distributions?, for which the Kolmogorov-Smirnov test is commonly used. The KS test statistic is ...
- 61
6
votes
Kullback–Leibler vs Kolmogorov-Smirnov distance
Another way of stating the same thing as the previous answer in more layman terms:
KL Divergence - Actually provides a measure of how big of a difference are two distributions from each other. As ...
- 209
6
votes
Accepted
advantage of euclidean distance for classification
That depends a lot on your use-case.
If you're working in a continuous space where all dimensions are properly scaled and relevant, then Euclidean is going to be a great choice for distance function.
...
- 404
6
votes
Accepted
A similarity measure with binary data: does this one have a name?
Your measure seems to resolve to a distance defined by Simpson. See A Survey of Binary Similarity and Distance Measures page 44, equation 45.
- 2,052
6
votes
Accepted
Distance metric for source code
You can check the two links below:
A comparison of code similarity analysers
Measuring Code Similarity in Large-scaled Code Corpora
At the third link below a similarity measure is proposed, which ...
5
votes
Accepted
How to distance and to MDS-plot objects according their complex shape
This may be only a partial answer because I don't think the plot that you expect is really what is in the data, especially the "parallelity and continuity" of the intermediate signals. I will ...
- 8,920
5
votes
Why is Euclidean distance not a good metric in high dimensions?
As an analogy, imagine a circle centred at the origin. Points are distributed evenly. Suppose a randomly-selected point is at (x1, x2). The Euclidean distance from the origin is ((x1)^2 + (x2)^2)^0.5
...
- 404
5
votes
Ask for suggestions on clustering methods on a large dataset with mixed types of variables
Similarly to the previous answers, most of the following answer of mine is not specific to SAS, as I use R. However, there is one exception to that - please see below. It seems that there exist ...
- 8,005
5
votes
Accepted
Online course about distance measures
Honestly I do not think that such narrowly focused course exists anywhere (online or offline), but there is an Encyclopedia of Distances book by Deza and Deza (2009, Springer) that you could check.
Tim♦
- 117k
5
votes
Accepted
square root missing in code?
The whole thread you linked, and the code you showed which was provided as an answer there, is in terms of Mahalanobis Distance squared, not Mahalanobis Distance. For certain purposes, it is ...
- 12.7k
5
votes
Accepted
Mahalanobis distance invariant against further elements/individuals?
It is only because you have a small sample size and thus a "poor" estimation of $S$, which may be greatly influenced by the new individuals. Try it out with $n < 10000$ and, as long as the new ...
- 414
5
votes
Accepted
Is there a version of the Mahalanobis distance for matrices?
No. There are metrics that try to build on a similar concept using Wishart distribution. I have seen papers in MRI imaging that use the metrics. See p.16 in this slide deck: https://earth.esa.int/c/...
- 56.8k
4
votes
Why does k-means clustering algorithm use only Euclidean distance metric?
I've read many interesting comments here, but let me add that Matlab's "personal" implementation of k-means supports 4 non-Euclidean distances [between data points and cluster centres]. The ...
Only top scored, non community-wiki answers of a minimum length are eligible