30
votes
Compute a cosine dissimilarity matrix in R
Many answers here are computationally inefficient, try this;
For cosine similarity matrix
...
- 600
19
votes
Accepted
Quantify the similarity of bags of words
Let me address this by describing the four maybe most common similarity metrics for bags of words and document (count) vectors in general, that is comparing collections of discrete variables.
Cosine ...
- 1,609
15
votes
Accepted
Is feature normalisation needed prior to computing cosine distance?
The definition of the cosine similarity is:
$$
\text{similarity} = \cos(\theta) = {\mathbf{A} \cdot \mathbf{B} \over \|\mathbf{A}\|_2 \|\mathbf{B}\|_2} = \frac{ \sum\limits_{i=1}^{n}{A_i B_i} }{ \...
- 4,603
14
votes
What are the difference between Dice, Jaccard, and overlap coefficients?
From the wikipedia page:
$$J=\frac{D}{2-D} \;\; \text{and}\;\; D=\frac{2J}{J+1}$$
where $D$ is the Dice Coefficient and $J$ is the Jacard Index.
In my opinion, the Dice Coefficient is more intuitive ...
- 1,436
13
votes
Accepted
Calculate Earth Mover's Distance for two grayscale images
Having looked into it a little more than at my initial answer: it seems indeed that the original usage in computer vision, e.g. Peleg et al. (1989), simply matched between pixel values and totally ...
- 25.2k
13
votes
Accepted
Kolmogorov-Smirnov Test in Python weird result and interpretation
You got a couple of things wrong while reading the documentation of the Kolmogorov-Smirnov test.
First you need to use the cumulative distribution function (CDF), not the probability density function (...
- 10.7k
10
votes
Accepted
Are time series motifs and the Matrix profile algorithm a good fit for my problem?
Yes, the Matrix Profile allows discord discovery, which is very competitive for anomaly detection (according to multiple independent test)
And yes, while "finding similarities among time series" is a ...
- 331
10
votes
Kolmogorov-Smirnov Test in Python weird result and interpretation
In addition to the coding mistakes addressed in the other answer, there are two statistics mistakes in the post that I want to address.
If the p-Value is higher than my chosen alpha (5%) my samples ...
- 67.1k
9
votes
Accepted
Clustering with Latent dirichlet allocation (LDA): Distance Measure
LDA does not have a distance metric
The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it ...
- 3,602
8
votes
Accepted
What is the purpose of row normalization
This is a relatively old thread but I recently encountered this issue in my work and stumbled upon this discussion. The question has been answered but I feel that the danger of normalizing the rows ...
- 520
8
votes
Does Mercer's theorem work in reverse?
Does Mercer's theorem work in reverse?
Not in all cases.
Wikipedia: "In mathematics, specifically functional analysis, Mercer's theorem is a representation of a symmetric positive-definite function ...
- 2,110
8
votes
Normalizing Euclidean distance by the length of the vectors
If you want to calculate the Euclidean distance of normalized vectors, use cosine similarity instead. They are proportional to each other and roughly equivalent.
- 141k
7
votes
Accepted
Measuring Information Content of unannotated terms in a corpus, avoiding -log(0)
The simplest most common way to avoid a 0 probability in word frequencies is the Lidstone smoothing
Which is basically, instead of using
$$p(w_i)=\frac{\#(w_i)}{\...
- 1,831
7
votes
Distance Metrics For Binary Vectors
In addition to Jaccard and Dice, I've had success with:
Cosine Similarity: $\text{cos}(\theta) = \frac{{\bf u} \cdot {\bf v}}{||{\bf u}|| \times ||{\bf v}||}$
Not a metric, only a similarity measure....
- 81
7
votes
What is the purpose of row normalization
There are various forms of row normalization and the OP is not stating which one s/he has in mind.
A specific form of row normalization (Eucledian norm normalization) where each row is normed (...
- 23k
7
votes
Comparison of two normal distribution
The question you are asking can be answered in multiple ways so hard to give a specific answer. However, if you are looking at the similarity between two normal distributions you could consider the ...
- 890
6
votes
What is the purpose of row normalization
Row normalization has a name -- ipsative scaling -- which typically involves rescaling a set of features by either dividing by the maximum value for the set or subtracting the mean of the features. ...
- 10.9k
6
votes
Most well-known set-similarity measures?
Other measures are:
Overlap Coefficient: $\frac{|A \cap B|}{min(|A|,|B|)}$
Tversky index: $|A\cap B| + \alpha|A\setminus B| + \beta|B \setminus A|$ where $\alpha$ and $\beta$ are positive numbers.
6
votes
Accepted
Can I apply word2vec to find document similarity?
Some time ago I tried this idea on 20 newsgroups data. I used GloVe embeddings from the authors site (Wikipedia ones).
Aggregating word embeddings using TF-IDF doesn't give good results. It is ...
- 5,986
6
votes
Accepted
How different will that be between the R-squared of linear regression y~x and square of cor(x,y)
Computationally, R-sq in computer printouts is the square $r^2$of the (Pearson) correlation $r.$ Sometimes $r^2$ is called the 'coefficient of determination'. ...
- 57.6k
6
votes
Accepted
Why does NMF of a symmetric matrix yield orthogonal matrices which are not transpose identical?
The reason why $H\neq W^T$ is that for two vector $a$ and $b$ to be orthogonal, you need:
$$\sum_{i=1}^n a_ib_i=0$$
This will typically be achieved by having $a_ib_i$ to be positive at some indices ...
- 2,690
6
votes
Accepted
Why do we not have a true upper limit for dissimilarity measure?
You are correct that dissimilarity is the opposite of similarity. However, it is not true that all similarity metrics are bounded between 0 and 1. For example, you can use dot product as a measure of ...
- 141k
6
votes
How to show that many functions (a hundred, a thousand) have the same shape an distribution of values over an interval?
To show that several functions are more or less the same you could just superimpose them graphically.
I don't think quantile plots of any flavour are directly relevant or likely to be helpful. The ...
- 59.5k
5
votes
Measuring Information Content of unannotated terms in a corpus, avoiding -log(0)
I think the appropriate answer will depend on what you want to use the "information content" for in the end.
I personally do not like the justification given for the second case, i.e. ...
- 13.1k
5
votes
Accepted
How to normalized a similarity matrix?
Assuming it's composed solely of positive values, and if your diagonal isn't already composed solely of ones, do:
$$A_{ij}:=\frac{A_{ij}}{\sqrt{A_{jj}\cdot A_{ii}}}$$
This is analogous to the ...
- 19.5k
5
votes
Accepted
Cosine similarity?
Let $x, y\in\{-1,+1\}^k$. Then their cosine similarity is
$$ \cos\theta =\frac{x\cdot y}{\|x\|_2\|y\|_2}=\frac{x\cdot y}{k} $$
since
$$ \|x\|_2=\|y\|_2=\sqrt{k}. $$
And
$$ x\cdot y = \#\{i\,|\,...
- 131k
5
votes
Accepted
Is there any well-founded way of calculating the euclidean distance between two images?
Consider that if we have two matrices of identical shape that we sometimes take their Frobenius norm which is essentially the kind of distance you describe. This would be adequate for black and white ...
- 9,660
4
votes
Accepted
If two graphs have similarity equal to 1 are their vertexes and edges equal?
Unfortunately, graph isomorphism is a harder problem then just matching vertex degrees, as this figure demonstrates.
Both are undirected graphs on the same six vertices, labeled "1" through "6". The ...
- 334k
4
votes
Alternate distance metrics for two time series
Answering question 1:
You critic of DTW is met by introducing global constraints to the warping path. This effectively restrains both computational effort (since warping paths which are not allowed ...
- 3,491
4
votes
Euclidean distance score and similarity
As you mentioned you know the calculation of Euclidence distance so I am explaining the second formula.
Euclidean formula calculates the distance, which will be smaller for people or items who are ...
- 41
Only top scored, non community-wiki answers of a minimum length are eligible