111
votes
Maximum Mean Discrepancy (distance distribution)
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\newcommand{\F}{\mathcal F}...
- 25.2k
30
votes
Accepted
How can this counterintiutive result with the Mahalanobis distance be explained?
"Why not draw a picture?" asks @mhdadk. Why not indeed?
Here are contours of the Mahalanobis distance/Gaussian likelihood centred at T (17, 4) (open ...
- 43.4k
28
votes
Accepted
Cosine Distance as Similarity Measure in KMeans
It should be the same, for normalized vectors cosine similarity and euclidean similarity are connected linearly. Here's the explanation:
Cosine distance is actually cosine similarity: $\cos(x,y) = \...
- 722
25
votes
What's the maximum value of Kullback-Leibler (KL) divergence
Or even with the same support, when one distribution has a much fatter tail than the other. Take
$$KL(P\vert\vert Q) = \int p(x)\log\left(\frac{p(x)}{q(x)}\right) \,\text{d}x$$
when
$$p(x)=\overbrace{\...
- 107k
22
votes
Accepted
What are the pros and cons of using mahalanobis distance instead of propensity scores in matching
Mahalanobis distance matching (MDM) and propensity score matching (PSM) are methods of doing the same thing, which is to find a subset of control units similar to treated units to arrive at a balanced ...
- 35.7k
19
votes
Accepted
Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error?
Overview:
KL-Divergence is derived from the Shannon entropy.
The Shannon entropy is the amount of information contained in a signal X with distribution $\mathrm{P}(X)$.
The cross entropy is the ...
- 12.1k
18
votes
Bottom to top explanation of the Mahalanobis distance?
I'd like to add a little technical information to Whuber's excellent answer. This information might not interest grandma, but perhaps her grandchild would find it helpful. The following is a bottom-to-...
- 92.6k
18
votes
How to measure the statistical "distance" between two frequency distributions?
You may be interested in the Earth mover's distance, also known as the Wasserstein metric. It is implemented in R (look at the emdist package) and in Python. We ...
- 129k
17
votes
What's the maximum value of Kullback-Leibler (KL) divergence
For distributions which do not have the same support, KL divergence is not bounded. Look at the definition:
$$KL(P\vert\vert Q) = \int_{-\infty}^{\infty} p(x)\ln\left(\frac{p(x)}{q(x)}\right) dx$$
...
- 694
16
votes
Accepted
Statistical significance of difference between distances
The question of "significantly" different always, always presupposes a statistical model for the data. This answer proposes one of the most general models that is consistent with the minimal ...
- 330k
14
votes
Which distance to use? e.g., manhattan, euclidean, Bray-Curtis, etc
Choosing the right distance is not an elementary task. When we want to make a cluster analysis on a data set, different results could appear using different distances, so it's very important to be ...
14
votes
Maximum Mean Discrepancy (distance distribution)
Here is how I interpretted MMD. Two distributions are similar if their moments are similar. By applying a kernel, I can transform the variable such that all moments (first, second, third etc.) are ...
- 141
13
votes
Accepted
Intuition of the Bhattacharya Coefficient and the Bhattacharya distance?
The Bhattacharyya coefficient is
$$
BC(h,g)= \int \sqrt{h(x) g(x)}\; dx
$$
in the continuous case. There is a good wikipedia article https://en.wikipedia.org/wiki/Bhattacharyya_distance. How to ...
- 81.7k
12
votes
Accepted
Comparing two histograms using Chi-Square distance
@Silverfish asked for an expansion of the answer by PolatAlemdar, which was not given, so I will try to expand on it here.
Why the name chisquare distance? The chisquare test for contingency tables ...
- 81.7k
11
votes
Coupling and Total variational distance
There are at least 2 ways to compute the total variation distance. The first is by using the definition of Total variation distance:
$TV(\mu,\nu)=\sup_{ A\in \mathcal{F}}\left|\mu(A)-\nu(A)\right|,$
...
- 4,299
11
votes
Is one-hot encoding and standardization of data equivalent to Gower's distance?
One hot encoding and then standardization puts much more weight on the categoricial variables. In particular, rare values will get a big distance. Gowers feels a bit more balanced to me. But in the ...
- 42.8k
11
votes
Accepted
Intuition behind Weight of Evidence and Information Value formula
It can be difficult to find sources giving precise definitions and good explanations of these concepts ... there is one R package at CRAN woe with a function ...
- 81.7k
10
votes
What is a practical explanation of affine equivariance and why does it matter for a covariance estimator?
I will first recall the property formally:
Given an $n$ by $p$, $n>p$ data matrix $X$, an affine equivariant estimator of location and scatter $(m(X), S(X))$ is one for which:
$$(0)\quad m(A X)=A ...
- 22.9k
10
votes
Accepted
What is the inverse square of a distance (Euclidean)?
Imagine that we want to classify as red or blue the unknown gray point in the data cloud. Your algorithm is set up to measure ...
- 26.7k
10
votes
Accepted
Do Autoencoders preserve distances?
No, they don't. We basically design them so that they cannot preserve distances. An autoencoder is a neural network which learns a "meaningful" representation of the input, preserving its "semantic" ...
- 18.3k
10
votes
Accepted
What is the intuitive difference between Wasserstein-1 distance and Wasserstein-2 distance?
The answer boils down to the difference between the distance and its squared value. Assume we use the Euclidean distance, and imagine we want to transform a histogram $P_S$ whose bins are $x\in \...
- 713
9
votes
Accepted
Similarity metrics for more than two vectors?
The cosine similarity between two column vectors $x_1$ and $x_2$ is simply the dot product between their unit vectors
$$\mathrm{CosSim}[x_1,x_2]=\frac{x_1}{\|x_1\|}\bullet\frac{x_2}{\|x_2\|}$$
and ...
- 13.1k
9
votes
Accepted
Clustering with Latent dirichlet allocation (LDA): Distance Measure
LDA does not have a distance metric
The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it ...
- 3,602
9
votes
How to use Gower's Distance with DBSCAN algorithm in Python
While gower distance hasn't been fully implemented into scikit-learn as a ready-to-use metric, we are lucky that many of the clustering-related functions (e.g., ...
- 91
9
votes
Accepted
Best practices in the selection of distance metric and clustering methods for gene expression data
This will probably not be the answer you want or expect, but this is how I see these things.
Clustering problem
Clustering, to a degree, is almost always a subjective procedure. You decide how you ...
- 5,104
9
votes
Multivariate Wasserstein metric for $n$-dimensions
Wasserstein in 1D is a special case of optimal transport. Both the R wasserstein1d and Python scipy.stats.wasserstein_distance ...
- 4,005
8
votes
Finding the best path through the matrix in DTW
You have presented a matrix showing the pointwise distance computed by using the squared Euclidean distance. Each element of this matrix will be referred to as ...
- 3,481
8
votes
Accepted
What is the purpose of row normalization
This is a relatively old thread but I recently encountered this issue in my work and stumbled upon this discussion. The question has been answered but I feel that the danger of normalizing the rows ...
- 520
8
votes
Intuition on the Kullback–Leibler (KL) Divergence
The textbook Elements of Information Theory gives us an example:
For example, if we knew the true distribution p of the random
variable, we could construct a code with average description length
H(p)....
- 6,830
8
votes
Does Mercer's theorem work in reverse?
Does Mercer's theorem work in reverse?
Not in all cases.
Wikipedia: "In mathematics, specifically functional analysis, Mercer's theorem is a representation of a symmetric positive-definite function ...
- 2,110
Only top scored, non community-wiki answers of a minimum length are eligible