Questions tagged [similarities]

Measures of closeness between distributions, clusterings, data sets or other objects.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Similarity measure for two lists of scores

I have a finite set of items, a score for each item, and a series of orderings: ...
0
votes
0answers
8 views

I don't understand the task of finding the correlation in WordSimilarity-353

My issue is the following, I have created word embeddings using WordNet, and to test my embeddings to see how they stack up against the word similarities in WordSimilarity-353 I'm running a script ...
1
vote
1answer
46 views

Rao-Stirling diversity index

I need help to calculate the Rao-Stirling diversity index. I tried it several times but cannot achieve equal results to R packages yet (e.g. diverse). I used the Rao-Stirling diversity index as ...
1
vote
0answers
13 views

Dynamic Time Warping and non-stationarity of time series

Dynamic Time Warping (DTW) computes the optimal alignment between points of two time series. If we consider two non-stationary time series (e.g. the first proportional to the second), is the DTW ...
0
votes
0answers
4 views

RBF-Kernel: Handling missing values

I want to compute the RBF-Kernel for a dataset which contains missing values: ...
1
vote
0answers
27 views

Distance measure for two probability distribution of unequal sample size

Context: I have 100 stores and these stores are divided into 10 business markets. I want to select 3 markets where each market is a good representation of the 100 stores i.e. the population. There ...
2
votes
1answer
116 views

Finding similarity between two datasets [closed]

I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state. ...
1
vote
0answers
17 views

Probability that output is result of the same process?

I've developed a simple Monte-Carlo simulation. Output of this simulation is a histogram. This histogram is possibly a log-normal distribution, but I don't want to assume that. But I do know that the ...
0
votes
1answer
16 views

Training classifier on randomly generated negative samples

I have $M$ (~dozen million) feature vectors. There are $F$ (~several dozen thousand) binary features, but in each vector, only $H$ (~several hundred) of them would be 1, the rest are 0. Now, for a ...
0
votes
0answers
15 views

Understanding formula for comparing players based on certain metrics

I'm looking at this tool: https://www.capfriendly.com/comparables/ and trying to understand the formula behind its rankings. I don't really need the exact formula, I really just need a way to compare ...
0
votes
1answer
41 views

Cosine similarity vs quadratic objective function

The aim is to calculate the similarity between two foods given the nutritional content of each. After some reading, it seems the most popular measure for this sort of problem is the cosine similarity ...
1
vote
1answer
80 views

FindSimilar items in a complex dataset

Im a MachineLearning newbie, but I want to learn more about this interesting topic using a practical example, on which I would appreciate any theoretical and practical help: I have a database of "...
0
votes
0answers
35 views

Weigh Kullback Leibler Divergence with P entropy?

I'm wondering if it makes sense to weight Kullback-Leibler Divergences to highlight divergences on highly distinguishing features. I am however not very well versed in Information Theory, and would ...
1
vote
0answers
36 views

Name of a similarity test for Ordinal data to see how similar a set of participant answers are to expert answers

I asked 12 participants in a small experiment to order 5 distorted webpages from the least to most trustworthy (1 - 5). I want to know the best way to test how similar participants answers are to a ...
1
vote
1answer
29 views

How do I statistically compare the readings from two different sensors? [duplicate]

I have temperature and humidity readings from two different sensors (located in the same geographical area). The data is logged every hour and I have data for more than a year, logged this way 2/1/...
0
votes
1answer
63 views

How to find similar users in a social network

I have a set of users from a social network. These users are represented by large sparse vectors. Let's say that a small subset of those users bought a ticket for a particular movie. How could I find ...
0
votes
1answer
36 views

What is the method should I use to calculate similarity in a data set with outliers that must be included?

Information: So I have a data set with 18 vectors with 167 components, each of with has a value with a range of $[-2, 2]$. I am trying to calculate the similarity between one arbitrary vector in ...
1
vote
1answer
209 views

Dynamic Time Warping vs Cross Correlation

Consider the following data frames: > df1 value 1 6 2 2 3 3 4 1 > df2 value 1 3 2 8 3 4 4 5 I want to find the ...
0
votes
1answer
67 views

multiple measurements for classification; which one to choose?

There are five different classes (labels) where the new points have to be classified at. For each new point, five different measurements are conducted, resulting into similar although different values ...
0
votes
0answers
16 views

Compute simliarity between two sets of numbers with each set summing up to 100%?

I have product shares for different users. Need to find similarity between users based on these product shares. For Example, User 1 Prod A: 44% Prod B: 33% Prod C: 23% User 2 Prod A: 76% Prod B: 24%...
0
votes
1answer
53 views

How to compare two bayesian linear models?

Let say, we have two bayesian linear models learned from some data (not necessarily same but from the same data distribution), how can we compare or what is the notion of similarity between the two ...
0
votes
1answer
213 views

Interpretation of clustering on similarity matrix

Many papers suggest doing clustering not on the (n x p)raw data but on the n x n or p x p matrices computed according to determined similarity measures (eg, correlation, cosine, point mutual ...
0
votes
0answers
56 views

Can Bhattacharyya coefficient (or distance) be used as an additive measure to compute a metric for performance?

As far as I understand, Bhattacharyya's measure(s) can be used to see similarity between two empirical distributions. Other ways to do so are nicely explained here: Similarity measure between multiple ...
0
votes
1answer
142 views

Is Hierarchical clustering a special case of knn(specific n=1)?

I'am working on time series in the scope of similarity detection at the moment. What seems to be a well researched approach is dynamic time warping in combination with k-1NN as classification ...
0
votes
0answers
10 views

Vector representation of subsets of data in python

I have 141575 data points that can be divided into 153 blocks. There are around 70 features in my dataset. I want to measure similarity among the blocks and rule out outliers during training my model. ...
2
votes
2answers
121 views

How different will that be between the R-squared of linear regression y~x and square of cor(x,y)

Generally, both of them can represent the linear relationship between x and y scale to [0,1]. Are they 99% very similar?
0
votes
0answers
31 views

Set Similarity Measure that accounts for size

Suppose I have two sets, A and B where 10|A| = |B| and A is a subset of B. For the case where |A| = 1 the Jaccard similarity of the sets will be 0.1. For the case where |A| = 100 the Jaccard ...
1
vote
1answer
38 views

Practical handling of Dynamic Time Warping for time-series with unequal sampling frequency (irregular time series)

Hi all and thank you for taking time to read this. When I read the general literature about dynamical time warping (and the vignette for the $\texttt{R}$ package $\texttt{dtw}$), the time series seem ...
0
votes
0answers
18 views

How to Measure/Quantify Similarity between > 2 Sets of Dummies Indicators

I have multiple sets, each with a binary indicator for whether an element is part of the set. ...
0
votes
2answers
53 views

Clustering high dimensional data

I was going through this wiki page on clustering in high dimensions and I don't understand the following statement there. Can someone explain to me what this means? The concept of distance becomes ...
0
votes
1answer
90 views

Similarity between n >> 2 binary vectors

I have $n$ binary vectors $x_i, ..., x_n$ of length $d$. I wish to compute a similarity measure between all $n$ of them. My initial thought was to use the Jaccard Index, since each binary element $x_{...
0
votes
0answers
34 views

Similarity of form of two curves

Currently I'm working on a project on finding out the similarity of two movies based on the sentiment analysis I did on their subtitles. I measured the sentiment of each individual sentence and paired ...
0
votes
1answer
129 views

Comparison of two normal distribution

I have two normally distributed samples. I want to know how close or similar it is. I tried few methods to find the similarity, like z-score and bhattacharyya distance. Bhattacharyya distance didn't ...
3
votes
1answer
561 views

Calculate Earth Mover's Distance for two grayscale images

I am trying to calculate EMD (a.k.a. Wasserstein Distance) for these two grayscale (299x299) images/heatmaps: Right now, I am calculating the histogram/distribution of both images. The histograms ...
2
votes
1answer
168 views

Statistically compare similarity between images

I have two images/heatmaps (2d matrix) of identical size. I need to statistically compare the similarity between the two. With 'similarity', I mean that high and low values of one image appear in ...
0
votes
1answer
18 views

similarity function for keywords that often occur together

I'm working on a project where I have to develop a machine learning solution. I should mention that I'm not very experienced in ML yet and that I'm not even sure if the term "similarity function" is ...
0
votes
0answers
16 views

Dimensionality of similarity matrix

Below is a screen shot of a paper. The authors take a data-set $E\in R^{nxm}$. Here $n$ is the number of observations/samples/patients and $m$ is the number of genes/features. Preprocessing ...
0
votes
1answer
36 views

What distance function to use for matching observations on a mix of ordinal, nominal and numeric variables?

I'm trying to match potential respondents to a random sample from the target population, and not sure whether any of the usual methods available out-of-the-box in e.g. R are right for the job. ...
0
votes
0answers
11 views

Term x term matrix for text clustering. What to do with subterms of n-grams

I am doing topic discovery on a large corpus. Reading here and there I found some papers saying that in case of big, sparse document x term matrices is better to create first a term x term similarity ...
0
votes
0answers
38 views

Clustering/Similarity between drivers

I have a dataset that contains initial and ending points of car trips: ...
1
vote
0answers
33 views

Is it possible to evaluate similarity (as in Kolmogorov-Smirnov test) between two sample vectors using copulas?

Consider the Two-sample Kolmogorov-Smirnov test: given 2 vectors of samples $X$ and $Y$ it is possible to test whether they are drawn from the same continuous distribution by calling (in R, for ...
2
votes
1answer
341 views

What GLM family and link function to use with similarity index as response variable?

I need to model an unusual response variable (at least to the best of my knowledge) that is: similarity (estimated by Morisita-Horn index) in species composition between pairs of sites. My response ...
0
votes
1answer
13 views

String similarity measure for pitch contour similarity (Parson code)

I have a dataset with short melodies hummed by experimental subjects. Each melody consists of a variable number of notes (10 to 20 notes) and is coded by the pitch levels of the notes (automatically ...
0
votes
0answers
21 views

Using the appropriate method for measuring the similarity of 1-D discrete time series data

I have a data of a 1-D time series data (more than 4000 points), they have fixed spacing, sample at fixed frequency. My task is to 'extract' the part in this data that match/look similar to my input ...
1
vote
0answers
34 views

R daisy - Gower distance, different values with different number of observations [duplicate]

I have a mixed data set where I want to compute distances between observations with Gower in R (daisy function). When I compute distances between different number of observations, the distances seem ...
4
votes
0answers
22 views

What can I use to compute a similarity (or diversity) index for a sample with “multidimensional” attributes?

Current problem: We have a batch of $n$ items for which we capture their details with $m$ attributes. It could look something like this: The goal is to compute an "index" that says how "similar" this ...
0
votes
0answers
7 views

How to comparing the similarity between two predictions with 2 fixed+1 alternative outcomes?

I want to compare the similarity of two diseases to Parkinson's disease (PD) with the same predictors. My first instinct is to run a multinomial regression with two fixed ends (healthy control and ...
0
votes
0answers
37 views

Similarity measure before and after dimensionality reduction or clustering

I have a dataset with 500 000 samples, each sample contains 30 features. The values of the features are in the range 0.0 to 1.0. ...
2
votes
1answer
63 views

Distance metric with characteristics of cosine and Manhattan

I'm working on a project where I want to find similarities between groups of events. So far I have expressed groups of events as vectors of event counts and computing similarities between them. I'm ...
0
votes
0answers
18 views

Mahalanobis: Covariance or correlation? [duplicate]

Is the Mahalanobis distance using correlations or covariances (among two vectors) to determine the similarity? I know that both are quite the same and differ only by a division of a standard deviation ...