Questions tagged [similarities]
Measures of closeness between distributions, clusterings, data sets or other objects.
504
questions
0
votes
0
answers
14
views
A common mathematical framework to introduce different "measures of distributional divergences/similarity"
Preface. I am going to ask here the same question ("A common mathematical framework to introduce different "measures of distributional divergences/similarity") that I asked on math....
- 168
0
votes
0
answers
16
views
How to test similarity between multiple medical differential diagnoses?
I have multiple observations collected by doctors where they give the differential diagnoses, the 10 most probable diseases (d). I want to calculate the inter-observer similarity.
my data looks like ...
- 1
0
votes
0
answers
31
views
Calculate the cosine similarity between two probability measures, rather then between two vectors
Introduction. The cosine similarity is often calculated among two vectors, i.e. $C_S(X,Y)$:
\begin{equation}
C_S(X,Y) = \frac{\sum_{i=1}^{n}(X_i,Y_i)}{\sqrt{\sum_{i=1}^{n}X_{i}^{2} \sum_{i=1}^{n}Y_{i}^...
- 168
0
votes
0
answers
13
views
How to adjust similarity scores by removing the influence of a common vector?
I have a similarity score function, $s(x,y)$. I know that I have two items that I'm trying to compare the similarity of, but both are based on the same template. How would I remove the template from ...
- 101
0
votes
0
answers
15
views
Timeline "spike" similarity metric?
Imagine you have a single axis. Along the axis are placed several target points, at differing distances between each other.
You now have a black box that shoots a variable number of points on to the ...
0
votes
1
answer
17
views
Specific correlation metric for categorical variables where matched values are of primary interest - does it have a name?
Hello fellow StackExchangers,
Let's say I have an experiment where two people are asked several questions, and for each question, they can independently select one of 5 preformed answers (A to E). So ...
- 3
1
vote
0
answers
9
views
Similarity metrics for circular and unaligned time series
I face the problem of calculating the similarity between multiple time series. What I want to do is compare each one with the first one to quantify the degree of change.
The challenge is that they are ...
1
vote
1
answer
50
views
Normalizing Euclidean distance by the length of the vectors [closed]
Suppose I have 4 vectors, the first 2 vectors are of length 4 and the last 2 vectors are of length 400. all values in the vectors range from 0.5 to 0.6.
The Euclidean distance between the last 2 ...
- 31
0
votes
1
answer
47
views
Correlation vs Euclidean distance as measures of similarity or closeness between data points with an outlier
I am interested in the comparison of Pearson correlation and Euclidean distance as measures of similarity between data points. Suppose I have 4 data points, w, x, y, z, in a multidimensional space, ...
-1
votes
1
answer
46
views
Apply divergences between two "relative frequency distributions", instead of between two "probability distributions"
Introduction. Recalling that:
The frequency is the number of observations of a specific outcome.
The relative frequency is a proportion of all observations (frequency / total observations).
A "...
- 168
0
votes
1
answer
35
views
Weighted entropy using similarity scores
I have a list (freq) of letters with the following frequencies:
...
- 83
2
votes
0
answers
11
views
Differential Privacy guarantee that takes into account the approximate density (e.g., the pseudo randomness) used in practice?
In theory the differential privacy guarantee comes from adding randomness to an algorithm so whatever is output is a sample from a target distribution (e.g., the Laplacian, Gaussian, Exponential ...
- 351
0
votes
0
answers
32
views
Does the number of bins influence the "measures of similarity" among histograms?
I have calculated several "measures of similarity" among histograms and discrete probability distributions (such as Cosine similarity, Kullback-Leibler divergence, Jensen-Shannon distance, ...
- 168
0
votes
0
answers
21
views
How to normalise dissimilarity and exposure indices (for longitudinal data)
I'm currently working with longitudinal data and I wanted to compare school and residential segregation (natives vs minorities) over time and across specific territorial sub-units (NUTS 3 level).
The ...
- 1
0
votes
1
answer
96
views
Sample size/ power estimation for similarity index (Dice/ F1 Score)
We will start a study in order to validate an algorithm for image segmentation for medical use. We will use Dice Sorensen coefficient in order to measure the agreement/ similarity between the ...
- 191
0
votes
1
answer
71
views
Finding trend similarity between two different time series
I am very new to this data analytics field, so bit confused about what to do.
I have 2 time series datasets of two different products that have the same attributes. I want to find trend similarity ...
- 101
0
votes
0
answers
31
views
Measuring similarity between two datasets
This is quite straightforward:
I have a dataset with 8 features containing cell measurements from a patient. I want to compare two patients to a reference one. To do this I perform UMAP to reduce the ...
- 13
0
votes
0
answers
53
views
Should similarity matrix must always be converted into a dissimilarity matrix for hierarchical clustering?
Background
Hi all, I need some clarification on my approach if it's correct or not. I have a matrix (M_ij) with user ratings of images. The users (i) are on the horizontal axis and the images (j) are ...
- 1
1
vote
0
answers
10
views
How to include a nested factor in a SIMPER test in PRIMER [closed]
I'm trying to conduct a SIMPER analysis in PRIMER. I'm wanting to do the analysis using a factor that is made up of the interaction between 2 other factors, with one nested in a third.
I'm looking at ...
- 11
2
votes
1
answer
76
views
How to compare two dendrograms
I want to find a method to compare two dendrograms. I have documents that were read by some people and they made a dendrogram that indicates which documents they think are more similar. Now I want to ...
- 21
0
votes
0
answers
12
views
Similarity measure over time
I'm completely new to this so I apologise in advance if my question isn't even phrased correctly.
I want to analyse changes in similarity of two interacting entities over time. Each entity is ...
0
votes
0
answers
20
views
Does this metric exist for clustering data based off its similarity matrix?
I'm working with a dataset where all I have is the similarity matrix (with values 0 to 1, 0 being no similarity, 1 being identical). After I assign the labels, I loop through the matrix and, for each ...
0
votes
0
answers
18
views
Similarity of two sets of scatter data
I am currently working on a project where I have a dataset of scatter-points in space. What I am doing is that I fit Gaussian Mixture Models (GMM) on the data set and use the model to sample '...
1
vote
0
answers
16
views
Measure "similarity" of different feature sets on the same samples
I have a population that I have quantitatively profiled using two different technologies. The result is two data matrices that have the exact same set of individuals as rows, but different features (...
- 8,051
1
vote
0
answers
45
views
Finding the nearest point to a given set of points [closed]
I am trying to build an auto-suggestion logic which looks at currently selected items and recommends a list of items to select the next item from.
I can formulate the problem thus:
Given a set of ...
0
votes
2
answers
114
views
How to measure the similarity of frequency of categorical data
Sorry if this is a really simple question but I'm new to this and wondered if there's an easy way to do what I'm picturing.
Imagine I've got a bunch of people and I'm asking them what they've eaten in ...
0
votes
0
answers
17
views
Matching items based on similarity
I would like to ask for a proposal for a machine learning model that would be suitable for the following problem:
I have a training set where each element of type A corresponds to a certain number of ...
- 31
0
votes
0
answers
35
views
Compare similarity or difference of two distributions by the ratio of moments
I'm looking for a measure that measure the similarity of two distributions in the following forms:
$$
S = \frac{a \mu_1 + b \sigma_1}{a \mu_2 + b \sigma_2}.
$$
The above formula I proposed is not ...
- 1
1
vote
1
answer
42
views
Similarity measure for two discrete distributions (a group of proportions)
I am not really sure how to describe it with the correct mathematical terms. If there are any questions, I will try to explain further.
I have several instances of a group of proportions. One could ...
1
vote
0
answers
24
views
Dimensionality reduction that preserves non-trivial similarity
I have a set of vectors $x_i \in\mathbb{R}^m$ and a similarity function f that quantifies how similar $x_i,x_j$ are. Unfortunately, calculating f takes a lot of time.
I want to use some neural ...
- 21
0
votes
0
answers
91
views
How to calculate the Jaccard index of co-occurence matrix without diagonal values?
I transformed a 2-mode matrix into a 1-mode co-occurence matrix and canceled all the values on the diagonal (because I thought it was meaningless). I wanted to use the jaccard method to calculate the ...
- 33
2
votes
0
answers
35
views
Comparing the similarity of ranked word lists with a reference standard
Imagine a scientific organisation wants to communicate the likelihood of particular events occurring. For consistency, the organisation decides to always draw from the same set of words, which it ...
- 21
1
vote
0
answers
52
views
R adjusted cosine similarity [closed]
I would like to find an effective way to get adjusted cosine similarity for a large matrix (10k rows).
Apply seems also slow...
Could you please suggest a method?
Maybe based on the approaches ...
- 11
3
votes
0
answers
39
views
Calculating similarity/resemblance of time series data? (soil moisture)
I have measured a variable (soil moisture) for multiple samples over time. Soil moisture increases steeply at several points in time (when it rains) and then decreases steadily until the next rain ...
- 31
1
vote
1
answer
35
views
Statistical argument for this cluster measure
This quick clustering score discussion presents the following single cluster scoring functions:
$$
c_i = 1-\sqrt{\frac{\sum_{j}^{N}\left(1-\phi(i,j) \right)^2}{N-1}}
$$
and
$$
C = 1-\sqrt{\frac{\sum_{...
- 123
0
votes
1
answer
83
views
How to compare two ranked lists of genes with ranking values?
I have 80 independent sets of genes with 6000 genes in each. Gene expression values are ranked and scaled from 0 to 1. So, a set looks like this:
Gene1 1
Gene2 0.98
Gene3 0.85
...
Gene5998 0.002
...
- 119
4
votes
1
answer
193
views
Why do we not have a true upper limit for dissimilarity measure?
The similarity measure between two attributes always falls in the range of $[0,1]$, why is this not true in the case of dissimilarity where the value falls in the range of $[0, \infty)$?
Can't the ...
0
votes
0
answers
59
views
How to best compute pairwise similarity vectors with additional redundancy consideration
My goal is to compute pairwise similarity between 700 regions (i.e., columns of Dataframe #1 described below); hence a 700x700 output matrix.
I have two dataframes I would like to consider in the ...
0
votes
0
answers
34
views
When should we expect WLS to be superior to OLS?
My question is motivated by this paper. In a nutshell, they are proposing an Diff-in-diff estimator that is estimated via WLS and should be superior to the classical one, estimated via OLS.
In this ...
- 345
2
votes
0
answers
182
views
What does the PCA similarity factor tell us?
I found the following formula for comparing time series
$$ S_{\mathrm{PCA}}^{\lambda}=\frac{\sum \limits_{i=1}^{k} \sum \limits_{j=1}^{k}\left(\lambda_{i}^{(1)} \lambda_{j}^{(2)}\right) \cos ^{2} \...
- 31
1
vote
0
answers
28
views
The probability of two measurements being identical when both have measurement errors
Statement:
Let's assume $ X_i =\{x_1,\ x_2,\ldots,\ x_i \}$
is a set of products that I am interested in their length.
I have two measurement devices that measure the length of these products in ...
- 11
0
votes
0
answers
38
views
1
vote
0
answers
93
views
Find similarity of two ranked lists that have weights
Suppose I have two lists with columns rank, app_name, weight that look like this:
...
- 111
0
votes
0
answers
23
views
How can I discover similar or resembling sets by a statistical method?
Let say,
I have such m datasets:
...
- 689
1
vote
0
answers
46
views
How to define a distance metric between nodes of an Origin-Destination matrix?
I have an Origin-Destination matrix expressing (weekly) flows of people between every couple of nodes (cities). The number of people traveling from city $i$ to city $j$ in a specific week is $OD_{ij}$....
- 11
0
votes
1
answer
67
views
How best to compare two processes to know if they perform statistically the same?
I have two processes, A and B. I have tested A and B separately on different tasks, and each performed at a 90% level on their tasks (say, text classification). Now, I pick a random set of 100 tasks ...
- 1
2
votes
2
answers
95
views
Similarity percentage based on attributes and their quantities
I have two sets that contain (partially overlapping) attributes in different quantities and I am looking for a method to compare the similarity between the sets based on the attributes and their ...
- 121
2
votes
1
answer
363
views
Is there any well-founded way of calculating the euclidean distance between two images?
I need to determine the distance between two images.
Supposing that we are dealing with images of the same size, I think that we can reduce this problem to the square root of the sum of the square of ...
- 678
2
votes
1
answer
48
views
How can one evaluate the similarity of non-independent data-sets
I have a set of time series from different runs of a computational model. I know that the computational model is highly historical in the sense that the state early in a trial heavily influences (in a ...
- 123
5
votes
1
answer
842
views
Measures to compare clustering partitions
What are the most used measures (coefficients) to compare two partitions of objects into clusters? I am speaking of validating the results of clustering, not of classifying; the measures known as ...
- 55.9k