# Tag Info

### Compute a cosine dissimilarity matrix in R

Many answers here are computationally inefficient, try this; For cosine similarity matrix ...
• 532

### Hierarchical clustering with mixed type data - what distance/similarity to use?

If you have stumbled upon this question and are wondering what package to download for using Gower metric in R, the cluster package has a function named daisy(), ...
• 7,803
Accepted

### Quantify the similarity of bags of words

Let me address this by describing the four maybe most common similarity metrics for bags of words and document (count) vectors in general, that is comparing collections of discrete variables. Cosine ...
• 1,521

### Distance Metrics For Binary Vectors

Seems like you're looking for either the Jaccard distance or the Dice dissimilarity. Jaccard distance: $1 - \frac{|A \cap B|}{|A \cup B|}$ Dice dissimilarity: $1 - \frac{2|A \cap B|}{|A| + |B|}$ ...
• 356

### What are the difference between Dice, Jaccard, and overlap coefficients?

From the wikipedia page: $$J=\frac{D}{2-D} \;\; \text{and}\;\; D=\frac{2J}{J+1}$$ where $D$ is the Dice Coefficient and $J$ is the Jacard Index. In my opinion, the Dice Coefficient is more intuitive ...
• 1,366
Accepted

### Kolmogorov-Smirnov Test in Python weird result and interpretation

You got a couple of things wrong while reading the documentation of the Kolmogorov-Smirnov test. First you need to use the cumulative distribution function (CDF), not the probability density function (...
• 2,166
Accepted

• 96.2k

### Machine Learning technique for learning string patterns

Could your problem be restated as wanting to discover the regular expressions that will match the strings in each category? This is a "regex generation" problem, a subset of the grammar induction ...
• 431
Accepted

### Clustering with Latent dirichlet allocation (LDA): Distance Measure

LDA does not have a distance metric The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it ...
• 3,032
Accepted

### Are time series motifs and the Matrix profile algorithm a good fit for my problem?

Yes, the Matrix Profile allows discord discovery, which is very competitive for anomaly detection (according to multiple independent test) And yes, while "finding similarities among time series" is a ...
• 311
Accepted

### Calculate Earth Mover's Distance for two grayscale images

Having looked into it a little more than at my initial answer: it seems indeed that the original usage in computer vision, e.g. Peleg et al. (1989), simply matched between pixel values and totally ...
• 22.2k
Accepted

### A person repeatedly selects the two most similar items out of three. How to model/estimate a perceptual distance between the items?

A good approach to this kind of problem can be found in section 4 of the paper The Bayesian Image Retrieval System, PicHunter by Cox et al (2000). The data is a set of integer outcomes $A_1, ..., A_N$...
• 6,620
Accepted

### A valid distance metric for high dimensional data

For high-dimensional data, shared-nearest-neighbor distances have been reported to work in Houle et al., Can Shared-Neighbor Distances Defeat the Curse of Dimensionality? Scientific and Statistical ...

### Quantifying similarity between two data sets

Area between 2 curves may give you the difference. Hence sum(nr-nf) (sum of all differences) will be an approximation of the area between 2 curves. If you want to ...
• 8,993
Accepted

### topic similarity semantic PMI between two words wikipedia

You might compute PMI using Wikipedia, as following: 1) Using Lucene to index a Wikipedia dump 2) Using Lucene API, it is straightforward to get: The number (N1) of documents containing word1 and ...
• 241

### What is the purpose of row normalization

There are some field-specific reasons to perform row normalization. In text analysis, it is quite common to represent a text with the histogram of the words it contains. Starting from the count of ...
• 9,316

### Does Mercer's theorem work in reverse?

Does Mercer's theorem work in reverse? Not in all cases. Wikipedia: "In mathematics, specifically functional analysis, Mercer's theorem is a representation of a symmetric positive-definite function ...
• 2,060

### Similarity measure between two variables

Qualifications It so happens that in the Iris data set the rows (as is this data set is usually presented) are values on four variables, all with the same dimensions and units. However, I will not ...
• 48.9k

### Compute a cosine dissimilarity matrix in R

The following function might be useful when working with matrices, instead of 1-d vectors: ...
• 61

### TF-IDF versus Cosine Similarity in Document Search

Xeon is right in what TF-IDF and cosine similarity are two different things. TF-IDF will give you a representation for a given term in a document. Cosine similarity will give you a score for two ...

### How to find weights for a dissimiliarity measure

This is a big issue in some areas of machine learning. I'm not as familiar with it as I'd like, but I think these should get you started. Dimensionality Reduction by Learning an Invariant Mapping (...
• 11.2k
Accepted

### A similarity measure with binary data: does this one have a name?

Your measure seems to resolve to a distance defined by Simpson. See A Survey of Binary Similarity and Distance Measures page 44, equation 45.
• 2,032
Accepted

### distance measure of two discrete probability histograms (distance between two vectors)

Jensen-Shannon distance is the 1st thing I'd consider. If you don't insist on having a "distance function", you can directly use Jensen–Shannon divergence, from which this distance is derived. JS ...
• 670