Questions tagged [jaccard-similarity]
Jaccard similarity (or jaccard coefficient) is a similarity function for computing the similarity between two sets
76
questions
1
vote
1
answer
30
views
How do these different expressions of the Jaccard similarity coefficient relate?
I am aware the Jaccard similarity coefficient, $J$, is classically defined as the cardinality of the intersection divided by the cardinality of the union of two sets, $A$ and $B$. I can easily ...
- 111
0
votes
0
answers
38
views
Is there a better alternative for the Jaccard index to obtain the relationship between binary indices?
I currently have data from $n$ sources. Each source sends a binary signal each minute. I have a register of $p$ minutes.
I have used Python to calculate the Jaccard indices of each signal as
...
- 141
0
votes
0
answers
13
views
Drawing the correct card from a deck
Saw this on an old western show.
A poker player believes the dealer is switching decks of cards.
The player has 5 cards in his hand and at the end of a hand, he smudges all of them with cigar ashes. ...
- 11
1
vote
1
answer
65
views
Recommender system - for single user
I am building a recommender system, in which for one system there will be only one user. So we cannot use something like user-user data. Recommendation is an item which contains 10-15 attributes ie ...
- 21
0
votes
0
answers
105
views
How to calculate the Jaccard index of co-occurence matrix without diagonal values?
I transformed a 2-mode matrix into a 1-mode co-occurence matrix and canceled all the values on the diagonal (because I thought it was meaningless). I wanted to use the jaccard method to calculate the ...
- 33
0
votes
1
answer
290
views
Intersection over union vs sensitivity
In the context of segmentation, what is the difference between IoU and sensitivity? It sounds to me like they describe the same formula in different contexts but I might be wrong.
When true ...
- 101
1
vote
2
answers
207
views
Accounting for 0 observations in presence/absence community data, PCoA/PCA: is this idea credible?
I have a solution in mind for this problem, but I'm not sure if it is defensible (which is why I'm asking you all!). I have a data frame where each row represents an individual site, each column ...
- 55
1
vote
0
answers
249
views
How to calculate euclidian distance from similarity matrix
I have a similarity matrix but can not use it in the k-mean as input. So is there any way to convert the similarity matrix to the euclidean distance matrix?
Edit: I have Jaccard, overlap, cosine,(...
1
vote
1
answer
340
views
About LSH (Locality-sensitive Hashing) and MinHashing implementation using jaccard
I'm trying to implement this paper
Browser Fingerprint Coding Methods Increasing the Effectiveness of User Identification in the Web Traffic
I got a couple of questions about the LHS algorithm in ...
- 121
0
votes
0
answers
134
views
Is the Bray-Curtis dissimilarity/similarity independent of richness/diversity?
I was told about potential issues with the Bray-Curtis dissimilarity/similarity measure. Specifically, I have been asked if Bray-Curtis is independent of richness/diversity. I am unsure how to answer ...
- 133
1
vote
0
answers
13
views
Clustering of individuals based on their time spent at various points of interest
Consider a set of individuals, denoted by $\mathcal{I}$, and a set of points of interest (POIs), denoted by $\mathcal{J}$ (with $|\mathcal{I} | \approx 10^4$ and $|\mathcal{J} | \leq 50$).
For each ...
- 111
1
vote
0
answers
114
views
How to determine agreement between two sets?
I have two sets of Ids both drawn from the same background. I'd like to know the appropriate way to compare their similarity. Set A and B are often significantly different in length
eg.
...
- 11
2
votes
1
answer
1k
views
Are Jaccard score and F1 score monotonically related?
I have compared the rankings obtained by comparing 10+ classifiers with this two metrics:
Jaccard score
F1 score
They show a perfect correlation. This results holds on 50+ datasets. When comparing ...
- 151
1
vote
0
answers
21
views
Clustering - Distance Metric for Comparing Short Lists of Terms (non-repeating, no frequency)
Clustering involves using some distance or similarity metric.
What is the best way to score the similarity of these small sets of words? Criteria: These are technical terms which are extracted from ...
- 363
0
votes
1
answer
586
views
String distance equations for fuzzy-matching person names
A reproducible example with a small bit of R code is available in this stackoverflow post (link so I dont need to re-type out the code). The fuzzytext library in R ...
- 207
3
votes
2
answers
4k
views
Is Jaccard similarity/distance suitable for non-binary, quantitative data?
I have a dataset with each row a country and 10 columns with numerical features like GDP,Electrcity consumption, GNI etc. I am trying to use distance metrics to find similarity between the countries ...
- 33
2
votes
1
answer
643
views
Difference between an interaction and combining two variables into one with a similarity measure?
Conceptually, what would be the difference between interacting two variables in a regression and combining these two variables into one via a similarity measure? I'm looking at top managers and boards ...
- 535
0
votes
0
answers
475
views
Jaccard/binary (dis)similarity calculation to multidimensional scaling analysis
I have n N x n dataset of features (n) and subjects (N) in which I am attempting to cluster into a lower-dimensional space via multi-dimensional scaling.
I'm confused about which MDS setup I need to ...
- 337
0
votes
0
answers
201
views
Is there a way calculate a gradient of dissimilarity across multiple sites using presence/absence data?
I’ve been trying to come up with a metric for species (dis)similarity for fish in a series based on presence or absence data that gives a value of 1 or 0 for each lake. From my research, it seems like ...
0
votes
1
answer
310
views
How to create a "similarity index" on a set of variables
I have a set of 12 variables that describe the dwelling characteristics. 10 of them are dummies (i.e questions like "does the dwelling have a bath?" or "is there crime in the area?"), while one is a 4-...
0
votes
1
answer
64
views
distance metric for student course schedules
I'm doing an exploratory clustering analysis of student course schedules at a college. Interpretability by humans is paramount: we're trying to inform future research questions and possibly ...
- 103
1
vote
1
answer
864
views
Jaccard Similarity with Binary Data
I have 5400 rows of data and 3211 columns of attributes.
The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity ...
- 11
-1
votes
1
answer
252
views
jaccard distance if one samples is a subset
I have a very basic question but I cannot find an answer (especially for a clustering situation). I am trying to do hierarchical clustering of samples using jaccard distances. One sample contains ...
- 1
1
vote
2
answers
146
views
Addressing "Unbalanced Features" or Feature Taxonomy for Nearest Neighbor / Similarity Calculations
The main question is how to address an imbalance in representation of feature "sets" when calculating similarity. I'll motivate with an example scenario:
Suppose we have objects described by a binary ...
- 163
2
votes
0
answers
141
views
0
votes
3
answers
4k
views
Can I use Manhattan distance on binary data for hierarchical clustering?
I understand that classically Jaccard and Hamming work best with binary data, but is there anything specifically wrong with using a Manhattan distance instead with the complete linkage function?
- 157
4
votes
1
answer
816
views
Estimate Jaccard similarity based on a sample
The Jaccard similarity of two sets, $A$ and $B$, is defined as: $Jaccard(A,B)=\frac{A\cap{B}}{A\cup{B}}$.
Say that I only have a sample of $P\%$ of each of the sets: $A'$ and $B'$. What would be a ...
- 275
0
votes
0
answers
393
views
Measure similar to Jaccard index but comparing more than two groups at a time
I am interested in looking at the amount of sharing between several samples (each represented as a vector of character strings), and to do pairwise comparisions, I have been using Jaccard indices. Is ...
- 101
1
vote
1
answer
7k
views
Calculating Jaccard similarity in R: all duo combinations for multiple groups and creating a matrix or cluster plotting with the index [closed]
I have been trying to compute Jaccard similarity index for all possible duo combinations for 7 communities and to create a matrix, or preferably Cluster plotting with the similarity index.
There are ...
- 131
1
vote
0
answers
888
views
How do I calculate the Jaccard distance for normal distributions?
I'm trying to replicate the results in this paper. I'm stuck at this part: specifically,
calculate Jaccard intersection of pdata1 & pdata2
Is there some neat ...
- 11
5
votes
1
answer
2k
views
What is the significance of the Jaccard similarity score?
I understand how to calculate the jaccard similarity , but never quite understood the logic behind why are we calculating it. How does it show the similarity between two sets? What relation exactly ...
- 303
1
vote
0
answers
103
views
Correlation of continuous-time series
I have two continuous time series of discrete events. The data from them are timestamps of occurences, e.g.
...
- 2,748
1
vote
1
answer
273
views
Measuring similarity of observations (non numeric)
I have a dataset of format :
day,measurement1,measurement2
1,a,b
1,a,c
1,f,s
2,a,b
2,a,c
2,f,g
3,a,d
3,a,q
3,f,s
In this example day1 is more similar with day2 ...
- 637
0
votes
0
answers
381
views
Calculating Similarity between two multiset
I need to find similarity between two multiset. For example,
$$A = \{a, a, b, c, d\}$$
$$B = \{a, b, b, d\}$$
I know that I can use Jaccard Index but it gives same weight for each element. As ...
1
vote
0
answers
600
views
Dummy data clustering: OK to apply Euclidian distance hierarchical clustering on Jaccard similarity matrix?
I created dummy variables (binary data) from categorical variables where I want to partition N subjects into multiple classes by some clustering method. I created a Jaccard similarity index matrix for ...
- 300
0
votes
0
answers
129
views
Similarity between bar plots (counts of categories)?
What are the best ways to compare (calculate similarity) between two bar plots? For example:
...
- 715
8
votes
3
answers
8k
views
Similarity measures for more than 2 variables
If I have two binary variables, I can determine the similarity of these variables quite easily with different similarity measures, e.g. with the Jaccard similarity measure:
$J = \frac{M_{11}}{M_{01} +...
- 1,198
1
vote
1
answer
539
views
Interpretation of Jaccard similarity for binary networks
Given two undirected networks with the same amount of nodes defined by $A_{ij}^1$ and $A_{ij}^2$, we take the unique unordered pairs $(i, j)$. After sorting them, we then have two binary vectors $\vec{...
- 90
1
vote
0
answers
102
views
How to statistically validate that one clustering technique is more effective than other on Binary data
Problem statement - Idea is to cluster students based on their assessment pattern and score.
Data = I have a data set that contains Binary data of students answers(Correct/incorrect) which i have ...
-1
votes
1
answer
64
views
How to form clusters when features are sets of string objects
I am building a movie recommender. But unlike common recommender from movielens data, features of my movies are sets like actor list, genre list, list of producers and writers etc.
I know I can use ...
- 101
1
vote
2
answers
584
views
Detecting misclassified data points using distance matrix
I have a data set of around 65,000 items from a public database that have each been assigned to one of around 1,800 classes, but unfortunately I am aware that quite a few are misclassified. My goals ...
- 153
6
votes
2
answers
4k
views
Jaccard similarity coefficient vs. Point-wise mutual information coefficient
Can you explain the difference between the Jaccard similarity coefficient and the pointwise mutual information (PMI) measure? It would be great if you could add a few examples.
- 173
8
votes
2
answers
14k
views
Accuracy vs Jaccard for multiclass problem
TL;DR For a multiclass problem, is Jaccard score the same as accuracy?
Update March 29, 2019
The wrong implementation in scikit-learn is now fixed with pull request #13151. Hooray!
P.S. The lesson ...
1
vote
0
answers
199
views
Threshold value for nodes similarity
I am working on an algorithm for community detection on complex systems, where those systems are modeled as graphs. To form community i need to find similarity between nodes.
I am using Jaccard ...
- 111
0
votes
0
answers
534
views
Jaccard coefficient (0011 and 1100 most similar?)
I try to compare different binary objects, lets say:
...
- 173
2
votes
2
answers
3k
views
How to compare a Ranked List?
I have 50 products. For each product, I identify 3 related products (1 top related, 2 (partially related, least related)) using similarity measures. I want to compare the ranked list generated by my ...
1
vote
2
answers
1k
views
Jaccard similarity of sets with approximate element matching
Consider two sets like this:
A = {'hello', 'world'} and B = {'Hello', 'helo', 'goodbye'}.
I want to measure the similarity between these two sets. There are two levels to this problem:
Comparing ...
- 2,160
3
votes
1
answer
4k
views
NMDS from Jaccard and Bray-Curtis identical. Is that a bad thing?
The dataset is ecological (species abundance), where I am calculating distances between 51 sampling sites, based on abundances of 3200 species. The species abundance values go up to 2700, and the rest ...
- 61
4
votes
0
answers
10k
views
Jaccard distance vs Levenshtein distance for fuzzy matching
My data is similar to the following data, but far bigger and more complex.
Apple
Banana
Those fruits
Tomato
Cocumber
These vegetables
I would like to get the ...
- 5,151
18
votes
1
answer
30k
views
What are the difference between Dice, Jaccard, and overlap coefficients? [closed]
I come across three different statistical measures to compare two sets, in particular to segmentation on images (e.g., comparing the similarity between the ground truth and the segmented result).
...
- 12.7k