24
votes
Compute a cosine dissimilarity matrix in R
Many answers here are computationally inefficient, try this;
For cosine similarity matrix
...
- 532
17
votes
Hierarchical clustering with mixed type data - what distance/similarity to use?
If you have stumbled upon this question and are wondering what package to download for using Gower metric in R, the cluster package has a function named daisy(), ...
- 7,803
17
votes
Accepted
Quantify the similarity of bags of words
Let me address this by describing the four maybe most common similarity metrics for bags of words and document (count) vectors in general, that is comparing collections of discrete variables.
Cosine ...
- 1,521
14
votes
Distance Metrics For Binary Vectors
Seems like you're looking for either the Jaccard distance or the Dice dissimilarity.
Jaccard distance:
$1 - \frac{|A \cap B|}{|A \cup B|}$
Dice dissimilarity:
$1 - \frac{2|A \cap B|}{|A| + |B|}$
...
- 356
13
votes
What are the difference between Dice, Jaccard, and overlap coefficients?
From the wikipedia page:
$$J=\frac{D}{2-D} \;\; \text{and}\;\; D=\frac{2J}{J+1}$$
where $D$ is the Dice Coefficient and $J$ is the Jacard Index.
In my opinion, the Dice Coefficient is more intuitive ...
- 1,366
13
votes
Accepted
Kolmogorov-Smirnov Test in Python weird result and interpretation
You got a couple of things wrong while reading the documentation of the Kolmogorov-Smirnov test.
First you need to use the cumulative distribution function (CDF), not the probability density function (...
- 2,166
11
votes
Accepted
Is feature normalisation needed prior to computing cosine distance?
The definition of the cosine similarity is:
$$
\text{similarity} = \cos(\theta) = {\mathbf{A} \cdot \mathbf{B} \over \|\mathbf{A}\|_2 \|\mathbf{B}\|_2} = \frac{ \sum\limits_{i=1}^{n}{A_i B_i} }{ \...
- 4,440
10
votes
Accepted
How to get a valid distance metric?
First of all, in many applications you do not need a distance metric, but a dissimilarity will be okay. So make sure that triangle inequality is needed.
In mathematics, triangle inequality is part of ...
- 39.9k
9
votes
Calculating Jaccard or other association coefficient for binary data using matrix multiplication
The above solution is not very good if X is sparse. Because taking !X will make a dense matrix, taking huge amount of memory and computation.
A better solution is to use formula Jaccard[i,j] = #...
- 91
9
votes
Compute a cosine dissimilarity matrix in R
You can use the cosine function from the lsa package:
http://cran.r-project.org/web/packages/lsa
- 5,936
9
votes
Accepted
Building the connection between cosine similarity and correlation in R
The answer is really right there in your linked articles. From the first, here are the formulae for cosine and correlation (lightly edited for brevity and clarity):
\begin{align}
{\rm CosSim}(x,y) ...
9
votes
Kolmogorov-Smirnov Test in Python weird result and interpretation
In addition to the coding mistakes addressed in the other answer, there are two statistics mistakes in the post that I want to address.
If the p-Value is higher than my chosen alpha (5%) my samples ...
- 31.3k
8
votes
Accepted
Similarity function with given properties
The function
$$ f\colon [0,1]\times[0,1]\to[0,1], \quad(x,y)\mapsto \frac{1}{4}x+\frac{1}{4}y+\frac{3}{4}(x-y)^2 $$
does what you want. Plus, it's positive, symmetric and definite ($x\neq y$ implies ...
- 96.2k
8
votes
Machine Learning technique for learning string patterns
Could your problem be restated as wanting to discover the regular expressions that will match the strings in each category? This is a "regex generation" problem, a subset of the grammar induction ...
- 431
8
votes
Accepted
Clustering with Latent dirichlet allocation (LDA): Distance Measure
LDA does not have a distance metric
The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it ...
- 3,032
8
votes
Accepted
Are time series motifs and the Matrix profile algorithm a good fit for my problem?
Yes, the Matrix Profile allows discord discovery, which is very competitive for anomaly detection (according to multiple independent test)
And yes, while "finding similarities among time series" is a ...
- 311
8
votes
Accepted
Calculate Earth Mover's Distance for two grayscale images
Having looked into it a little more than at my initial answer: it seems indeed that the original usage in computer vision, e.g. Peleg et al. (1989), simply matched between pixel values and totally ...
- 22.2k
7
votes
Accepted
A person repeatedly selects the two most similar items out of three. How to model/estimate a perceptual distance between the items?
A good approach to this kind of problem can be found in section 4 of the paper The Bayesian Image Retrieval System, PicHunter by Cox et al (2000). The data is a set of integer outcomes $A_1, ..., A_N$...
- 6,620
7
votes
Accepted
A valid distance metric for high dimensional data
For high-dimensional data, shared-nearest-neighbor distances have been reported to work in
Houle et al., Can Shared-Neighbor Distances Defeat the Curse of Dimensionality? Scientific and Statistical ...
- 39.9k
7
votes
Quantifying similarity between two data sets
Area between 2 curves may give you the difference. Hence sum(nr-nf) (sum of all differences) will be an approximation of the area between 2 curves. If you want to ...
- 8,993
7
votes
Accepted
topic similarity semantic PMI between two words wikipedia
You might compute PMI using Wikipedia, as following:
1) Using Lucene to index a Wikipedia dump
2) Using Lucene API, it is straightforward to get:
The number (N1) of documents containing word1 and ...
- 241
7
votes
What is the purpose of row normalization
There are some field-specific reasons to perform row normalization. In text analysis, it is quite common to represent a text with the histogram of the words it contains. Starting from the count of ...
- 9,316
7
votes
Does Mercer's theorem work in reverse?
Does Mercer's theorem work in reverse?
Not in all cases.
Wikipedia: "In mathematics, specifically functional analysis, Mercer's theorem is a representation of a symmetric positive-definite function ...
- 2,060
6
votes
Similarity measure between two variables
Qualifications
It so happens that in the Iris data set the rows (as is this data set is usually presented) are values on four variables, all with the same dimensions and units. However, I will not ...
- 48.9k
6
votes
Compute a cosine dissimilarity matrix in R
The following function might be useful when working with matrices, instead of 1-d vectors:
...
- 61
6
votes
TF-IDF versus Cosine Similarity in Document Search
Xeon is right in what TF-IDF and cosine similarity are two different things. TF-IDF will give you a representation for a given term in a document. Cosine similarity will give you a score for two ...
- 529
6
votes
How to find weights for a dissimiliarity measure
This is a big issue in some areas of machine learning. I'm not as familiar with it as I'd like, but I think these should get you started.
Dimensionality Reduction by Learning an Invariant Mapping (...
- 11.2k
6
votes
Accepted
A similarity measure with binary data: does this one have a name?
Your measure seems to resolve to a distance defined by Simpson. See A Survey of Binary Similarity and Distance Measures page 44, equation 45.
- 2,032
6
votes
Accepted
distance measure of two discrete probability histograms (distance between two vectors)
Jensen-Shannon distance is the 1st thing I'd consider. If you don't insist on having a "distance function", you can directly use Jensen–Shannon divergence, from which this distance is derived.
JS ...
- 670
6
votes
Machine Learning technique for learning string patterns
You could try recurrent neural networks, where your input is a sequence of the letters in the word, and your output is a category. This fits your requirement such that you don't hand code any features....
- 2,078
Only top scored, non community-wiki answers of a minimum length are eligible