Questions tagged [similarities]
Measures of closeness between distributions, clusterings, data sets or other objects.
55
questions
30
votes
1
answer
24k
views
Converting similarity matrix to (euclidean) distance matrix
In Random forest algorithm, Breiman (author) constructs similarity matrix as follows:
Send all learning examples down each tree in the forest
If two examples land in the same leaf increment ...
- 487
54
votes
2
answers
43k
views
Hierarchical clustering with mixed type data - what distance/similarity to use?
In my dataset we have both continuous and naturally discrete variables. I want to know whether we can do hierarchical clustering using both type of variables. And if yes, what distance measure is ...
- 6,344
36
votes
1
answer
30k
views
Comparing hierarchical clustering dendrograms obtained by different distances & methods
[The initial title "Measurement of similarity for hierarchical clustering trees" was later changed by @ttnphns to better reflect the topic]
I am performing a number of hierarchical cluster analyses ...
- 2,192
20
votes
1
answer
25k
views
What is the optimal distance function for individuals when attributes are nominal?
I do not know which distance function between individuals to use in case of nominal (unordered categorical) attributes.
I was reading some textbook and they suggest Simple Matching function but some ...
- 361
50
votes
6
answers
54k
views
Percentage of overlapping regions of two normal distributions
I was wondering, given two normal distributions with $\sigma_1,\ \mu_1$ and $\sigma_2, \ \mu_2$
how can I calculate the percentage of overlapping regions of two distributions?
I suppose this problem ...
- 603
27
votes
2
answers
30k
views
Similarity Coefficients for binary data: Why choose Jaccard over Russell and Rao?
From Encyclopedia of Statistical Sciences I understand that given $p$ dichotomous (binary: 1=present; 0=absent) attributes (variables), we can form a contingency table for any two objects i and j of a ...
- 465
6
votes
1
answer
1k
views
Measures to compare clustering partitions
What are the most used measures (coefficients) to compare two partitions of objects into clusters? I am speaking of validating the results of clustering, not of classifying; the measures known as ...
- 57.5k
19
votes
3
answers
19k
views
Can someone please explain dynamic time warping for determining time series similarity?
I am trying to grasp the dynamic time warping measure for comparing time series together. I have three time series datasets like this:
...
- 4,532
16
votes
3
answers
25k
views
Distance Metrics For Binary Vectors
I have vectors of same length consisting of 1 and 0. I am trying to find out how similar they are. So far I am using hamming distance that I calculate sum of one vector then sum of second vector and ...
- 281
9
votes
3
answers
24k
views
Calculating Jaccard or other association coefficient for binary data using matrix multiplication
I want to know if is there any possible way to calculate Jaccard coefficient using matrix multiplication.
I used this code
...
- 319
7
votes
1
answer
3k
views
When to use dot-product as a similarity metric
I'm trying to understand which similarity measure should be used in which situations.
Roughly speaking, when should someone use the dot product to assess similarity between vectors?
Roughly speaking, ...
- 718
4
votes
1
answer
468
views
Measures to compare classification partitions
What are the most used measures (coefficients) to compare two partitions of objects into classes? I am speaking of validating the results of classification, not of clustering; the measures known as ...
- 57.5k
25
votes
5
answers
51k
views
Similarity measures between curves?
I would like to compute the measure of similarity between two ordered sets of points---the ones under User compared with the ones under Teacher:
The points are curves in 3D space, but I was thinking ...
- 351
17
votes
2
answers
15k
views
Alternate distance metrics for two time series
I have time-series data of different houses. Assume it is power consumption data. Now, I want to cluster the houses following similar power consumption pattern utmost. So, the various distance metrics ...
- 529
11
votes
2
answers
846
views
Does Mercer's theorem work in reverse?
A colleague has a function $s$ and for our purposes it is a black-box. The function measures the similarity $s(a,b)$ of two objects.
We know for sure that $s$ has these properties:
The similarity ...
- 91k
28
votes
5
answers
49k
views
Compute a cosine dissimilarity matrix in R [closed]
I want to create heatmaps based upon cosine dissimilarity.
I'm using R and have explored several packages, but cannot find a function to generate a standard cosine dissimilarity matrix. The built-in <...
- 435
13
votes
2
answers
28k
views
Similarity measure between multiple distributions
To compare distributions, it is common to use box blots.
I'm looking for a similarity measure that calculates whether distributions are
the similar or not.
Ideally, given that e.g. four ...
- 151
12
votes
1
answer
2k
views
Are time series motifs and the Matrix profile algorithm a good fit for my problem?
I have huge multivariate time series to analyze (Terabytes of data) and I need fast, scalable algorithms for mainly two tasks:
finding similar patterns among time series. For example, imagine I ...
- 18k
10
votes
3
answers
11k
views
Calculate Earth Mover's Distance for two grayscale images
I am trying to calculate EMD (a.k.a. Wasserstein Distance) for these two grayscale (299x299) images/heatmaps:
Right now, I am calculating the histogram/distribution of both images. The histograms ...
- 165
7
votes
3
answers
4k
views
How to get a valid distance metric?
I have got a problem to devise a distance metric to get the similarity measurement of vectors. Someone suggested me to use dot product, which seems to me the same as the Cosine similarity metric; ...
- 1,739
7
votes
2
answers
11k
views
distance measure of two discrete probability histograms (distance between two vectors)
I have multiple sets of discrete probability histograms(vectors) and I want to measure the distance between each histogram. I have done some research but I am in doubt.
Literature suggest I could ...
- 73
5
votes
1
answer
2k
views
Turn a distance measure into a kernel function
I have read here that an easy way to turn a distance function $d$ into
a similarity function $s$ is to compute: $s = e^{-\gamma * d}$. I believe that
this is also what is done with the RBF kernel. ...
- 237
5
votes
1
answer
3k
views
Combining multiple similarity measures in hyperspectral images?
I have a hyperspectral image where the pixels are 21 channels.
So each pixel $\in \mathbb{R}^{21}$. I want to perform clustering on the pixels with similarity defined by two different measures, one ...
- 153
5
votes
2
answers
5k
views
Building the connection between cosine similarity and correlation in R
According to some articles (e.g. here) correlation is just a centered version of cosine similarity. I use the following code to calculate the cosine similarity matrix of the column vectors of a matrix ...
- 6,156
18
votes
3
answers
38k
views
Quantifying similarity between two data sets
Summary: Trying to find the best method summarize the similarity between two aligned data sets of data using a single value.
Details:
My question is best explained with a diagram. The graphs below ...
- 301
17
votes
0
answers
5k
views
How to compare two distance matrices?
Suppose that I have two distance matrices for the same set of items.
By a distance matrix I mean a square matrix whose (i,j)th entry holds the distance (in terms of cosine similarity) between ith and ...
- 453
10
votes
0
answers
10k
views
Properties of Levenshtein, N-Gram, cosine and Jaccard distance coefficients - in sentence matching
Let's say I have two strings:
string A: 'I went to the cafeteria and bought a sandwich.'
string B: 'I heard the cafeteria is serving roast-beef sandwiches today'.
...
- 541
10
votes
1
answer
1k
views
What's an intuitive way to understand how KL divergence differs from other similarity metrics?
The general intuition I have seen for KL divergence is that it computes the difference in expected length sampling from distribution $P$ with an optimal code for $P$ versus sampling from distribution $...
- 349
9
votes
3
answers
2k
views
Distance metric and curse of dimensions
Some where I read a note that if you have many parameters $(x_1, x_2, \ldots, x_n)$ and you try to find a "similarity metric" between these vectors, you may have a "curse of dimensioality". I believe ...
- 2,061
9
votes
2
answers
8k
views
similarity measure between two different ordered sequences
I know we can quantify the similarity between two sequences with the same length and same elements by rank order correlation. But how to measure similarity between two sequences of different length, ...
- 93
8
votes
1
answer
8k
views
Most well-known set-similarity measures?
I know of the Jaccard index and the Sørensen-Dice coefficient for computing set similarity, but have been unable to find any other algorithms related to set similarity. This site contains quite a few ...
- 646
8
votes
3
answers
8k
views
Similarity measures for more than 2 variables
If I have two binary variables, I can determine the similarity of these variables quite easily with different similarity measures, e.g. with the Jaccard similarity measure:
$J = \frac{M_{11}}{M_{01} +...
- 1,218
6
votes
1
answer
3k
views
Determining similar users from hierarchical clustering
I use hierarchical clustering to cluster users which are similar to each other based on a Jaccard coefficient.
I have now coded a solution to extract similar users based on hierarchical clustering: ...
- 637
5
votes
1
answer
6k
views
How to get Bhattacharyya Distance in Excel (or Matlab, or R)
I have several histograms which I would like to compare to one reference histogram to see which one is the most similar to the reference in terms of the shape of the distribution.
Kologorov-Smirnov ...
5
votes
1
answer
6k
views
Kullback-Leibler divergence of two normal distributions
I was recently trying to find a way to compute the KL-divergence between 2 populations that are normally distributed using the mean and variance of each population. But I found several different ...
- 465
5
votes
2
answers
2k
views
How to measure similarity of bivariate probability distributions?
I have three different distributions of 2D data:
or
Now I like to know whether distribution two is more similar to distribution one (2 to 1) than distribution three is to distribution one (3 to 1)? ...
- 51
5
votes
1
answer
1k
views
How to use Matrix Profile for dimension reduction and clustering
Matrix Profile (MP) has been used for clustering time-series segments. In the slideshow tutorial featured on the MP website, they use a figure to demonstrate projecting segment similarity onto an M-...
- 657
5
votes
2
answers
3k
views
Combining multiple similarity measures
I have a training set with $N$ instances $\{I_1,...I_N\}$,
where each pair of instances is associated with a similarity score $S(I_x,I_y)\in [0,1]$ indicates if the two instances are similar or not.
...
- 153
4
votes
1
answer
2k
views
Computing Jensen-Shannon Divergence between discrete and continuous distribution
Is it possible to compute the Jensen-Shannon divergence between a discrete and a continuous probability distribution, e.g. between a standard normal distribution and a distribution taking the values 1,...
- 475
4
votes
2
answers
2k
views
Comparing undirected weighted graphs
I want to get a sense of the similarity between weighted, undirected graphs.
My data are from an EEG experiment, where every vertex is an electrode, and every edge is the connectivity between two ...
- 51
4
votes
1
answer
2k
views
A similarity measure with binary data: does this one have a name?
There are many binary similarity measures (e.g. Jaccard, Sorensen, etc), each of them is sensitive to different properties of the compared sets. I would like to use the metric $S=\frac{N_{A\bigcap B}}{...
- 153
3
votes
2
answers
1k
views
Similarity between objects based on tags (binary features)
I have five millions of objects each of them having one or more tags. How do I compute statistically sound similarity score between each pair of the objects taking into account that:
There are 100 ...
- 389
2
votes
1
answer
2k
views
how to measure similarity of two datasets (matrices) of different length
There are related questions being asked already but my problem is i can't find a good method of measuring similarity between two datasets that are represented by various lengths of matrices. For ...
- 21
2
votes
0
answers
809
views
Covariance function with circular similarity property
My aim is to fit functions to covariance matrices. Furthermore I would like to have these functions positive definite.
For example the figure below shows a fitted covariance matrix modeled using a ...
- 438
2
votes
0
answers
637
views
How does the VAT algorithm work?
I want to know how the VAT algorithm for cluster tendency works in detail. As you can see in the picture below, R is the dissimilarity matrix and R-tilda is the ordered dissimilarity matrix. What is ...
- 21
2
votes
2
answers
2k
views
how to measure similarity of two lists of continuous data with different length
I have two lists of continuous data with different length.
a) How should I measure (dis)similarity of these two lists? or, as these lists can be formed into histogram, how can I quantify (dis)...
- 55
2
votes
1
answer
8k
views
kernels and similarity (in R)
I am trying fit different kernels to calculate similarity matrix in R. Here is example data - X matrix :
...
- 3,653
2
votes
0
answers
1k
views
Similarity metric for 2 sets of vectors
I'm trying to determine the similarity between two sentences.
I have vectors for each word in a corpus, and using cosine distance of the two vectors, I can get quite a good "similarity" score ...
- 121
2
votes
0
answers
224
views
Association between two asymmetric matrices
I have two matrices which elements are distances, created from an anisotropic cost analysis algorithm, between sites/populations.
...
- 195
1
vote
1
answer
248
views
What methods exist to measure how similar a set of sampled distributions are to each other?
I have a set of distributions that are sampled. How do I measure their similarity between themselves or as a group/set?
So, given a finite domain for a sampled function$F_j(x)$, with $x_i \in {1,\...
- 1,675