24 votes

Compute a cosine dissimilarity matrix in R

Many answers here are computationally inefficient, try this; For cosine similarity matrix ...
user avatar
  • 532
17 votes

Hierarchical clustering with mixed type data - what distance/similarity to use?

If you have stumbled upon this question and are wondering what package to download for using Gower metric in R, the cluster package has a function named daisy(), ...
user avatar
  • 7,803
17 votes
Accepted

Quantify the similarity of bags of words

Let me address this by describing the four maybe most common similarity metrics for bags of words and document (count) vectors in general, that is comparing collections of discrete variables. Cosine ...
user avatar
  • 1,521
14 votes

Distance Metrics For Binary Vectors

Seems like you're looking for either the Jaccard distance or the Dice dissimilarity. Jaccard distance: $1 - \frac{|A \cap B|}{|A \cup B|}$ Dice dissimilarity: $1 - \frac{2|A \cap B|}{|A| + |B|}$ ...
user avatar
  • 356
13 votes

What are the difference between Dice, Jaccard, and overlap coefficients?

From the wikipedia page: $$J=\frac{D}{2-D} \;\; \text{and}\;\; D=\frac{2J}{J+1}$$ where $D$ is the Dice Coefficient and $J$ is the Jacard Index. In my opinion, the Dice Coefficient is more intuitive ...
user avatar
  • 1,366
13 votes
Accepted

Kolmogorov-Smirnov Test in Python weird result and interpretation

You got a couple of things wrong while reading the documentation of the Kolmogorov-Smirnov test. First you need to use the cumulative distribution function (CDF), not the probability density function (...
user avatar
  • 2,166
11 votes
Accepted

Is feature normalisation needed prior to computing cosine distance?

The definition of the cosine similarity is: $$ \text{similarity} = \cos(\theta) = {\mathbf{A} \cdot \mathbf{B} \over \|\mathbf{A}\|_2 \|\mathbf{B}\|_2} = \frac{ \sum\limits_{i=1}^{n}{A_i B_i} }{ \...
user avatar
  • 4,440
10 votes
Accepted

How to get a valid distance metric?

First of all, in many applications you do not need a distance metric, but a dissimilarity will be okay. So make sure that triangle inequality is needed. In mathematics, triangle inequality is part of ...
user avatar
9 votes

Calculating Jaccard or other association coefficient for binary data using matrix multiplication

The above solution is not very good if X is sparse. Because taking !X will make a dense matrix, taking huge amount of memory and computation. A better solution is to use formula Jaccard[i,j] = #...
user avatar
9 votes

Compute a cosine dissimilarity matrix in R

You can use the cosine function from the lsa package: http://cran.r-project.org/web/packages/lsa
user avatar
  • 5,936
9 votes
Accepted

Building the connection between cosine similarity and correlation in R

The answer is really right there in your linked articles. From the first, here are the formulae for cosine and correlation (lightly edited for brevity and clarity): \begin{align} {\rm CosSim}(x,y) ...
user avatar
9 votes

Kolmogorov-Smirnov Test in Python weird result and interpretation

In addition to the coding mistakes addressed in the other answer, there are two statistics mistakes in the post that I want to address. If the p-Value is higher than my chosen alpha (5%) my samples ...
user avatar
  • 31.3k
8 votes
Accepted

Similarity function with given properties

The function $$ f\colon [0,1]\times[0,1]\to[0,1], \quad(x,y)\mapsto \frac{1}{4}x+\frac{1}{4}y+\frac{3}{4}(x-y)^2 $$ does what you want. Plus, it's positive, symmetric and definite ($x\neq y$ implies ...
user avatar
8 votes

Machine Learning technique for learning string patterns

Could your problem be restated as wanting to discover the regular expressions that will match the strings in each category? This is a "regex generation" problem, a subset of the grammar induction ...
user avatar
8 votes
Accepted

Clustering with Latent dirichlet allocation (LDA): Distance Measure

LDA does not have a distance metric The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it ...
user avatar
  • 3,032
8 votes
Accepted

Are time series motifs and the Matrix profile algorithm a good fit for my problem?

Yes, the Matrix Profile allows discord discovery, which is very competitive for anomaly detection (according to multiple independent test) And yes, while "finding similarities among time series" is a ...
user avatar
8 votes
Accepted

Calculate Earth Mover's Distance for two grayscale images

Having looked into it a little more than at my initial answer: it seems indeed that the original usage in computer vision, e.g. Peleg et al. (1989), simply matched between pixel values and totally ...
user avatar
  • 22.2k
7 votes
Accepted

A person repeatedly selects the two most similar items out of three. How to model/estimate a perceptual distance between the items?

A good approach to this kind of problem can be found in section 4 of the paper The Bayesian Image Retrieval System, PicHunter by Cox et al (2000). The data is a set of integer outcomes $A_1, ..., A_N$...
user avatar
  • 6,620
7 votes
Accepted

A valid distance metric for high dimensional data

For high-dimensional data, shared-nearest-neighbor distances have been reported to work in Houle et al., Can Shared-Neighbor Distances Defeat the Curse of Dimensionality? Scientific and Statistical ...
user avatar
7 votes

Quantifying similarity between two data sets

Area between 2 curves may give you the difference. Hence sum(nr-nf) (sum of all differences) will be an approximation of the area between 2 curves. If you want to ...
user avatar
  • 8,993
7 votes
Accepted

topic similarity semantic PMI between two words wikipedia

You might compute PMI using Wikipedia, as following: 1) Using Lucene to index a Wikipedia dump 2) Using Lucene API, it is straightforward to get: The number (N1) of documents containing word1 and ...
user avatar
  • 241
7 votes

What is the purpose of row normalization

There are some field-specific reasons to perform row normalization. In text analysis, it is quite common to represent a text with the histogram of the words it contains. Starting from the count of ...
user avatar
  • 9,316
7 votes

Does Mercer's theorem work in reverse?

Does Mercer's theorem work in reverse? Not in all cases. Wikipedia: "In mathematics, specifically functional analysis, Mercer's theorem is a representation of a symmetric positive-definite function ...
user avatar
  • 2,060
6 votes

Similarity measure between two variables

Qualifications It so happens that in the Iris data set the rows (as is this data set is usually presented) are values on four variables, all with the same dimensions and units. However, I will not ...
user avatar
  • 48.9k
6 votes

Compute a cosine dissimilarity matrix in R

The following function might be useful when working with matrices, instead of 1-d vectors: ...
user avatar
  • 61
6 votes

TF-IDF versus Cosine Similarity in Document Search

Xeon is right in what TF-IDF and cosine similarity are two different things. TF-IDF will give you a representation for a given term in a document. Cosine similarity will give you a score for two ...
user avatar
6 votes

How to find weights for a dissimiliarity measure

This is a big issue in some areas of machine learning. I'm not as familiar with it as I'd like, but I think these should get you started. Dimensionality Reduction by Learning an Invariant Mapping (...
user avatar
6 votes
Accepted

A similarity measure with binary data: does this one have a name?

Your measure seems to resolve to a distance defined by Simpson. See A Survey of Binary Similarity and Distance Measures page 44, equation 45.
user avatar
  • 2,032
6 votes
Accepted

distance measure of two discrete probability histograms (distance between two vectors)

Jensen-Shannon distance is the 1st thing I'd consider. If you don't insist on having a "distance function", you can directly use Jensen–Shannon divergence, from which this distance is derived. JS ...
user avatar
  • 670
6 votes

Machine Learning technique for learning string patterns

You could try recurrent neural networks, where your input is a sequence of the letters in the word, and your output is a category. This fits your requirement such that you don't hand code any features....
user avatar
  • 2,078

Only top scored, non community-wiki answers of a minimum length are eligible