We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [jaccard-similarity]

Jaccard similarity (or jaccard coefficient) is a similarity function for computing the similarity between two sets

Filter by
Sorted by
Tagged with
9
votes
1answer
8k views

What are the difference between Dice, Jaccard, and overlap coefficients? [closed]

I come across three different statistical measures to compare two sets, in particular to segmentation on images (e.g., comparing the similarity between the ground truth and the segmented result). ...
7
votes
4answers
4k views

Jaccard Similarity - From Data Mining book - Homework problem

Exercise 3.1.3 : Suppose we have a universal set U of n elements, and we choose two subsets S and T at random, each with m of the n elements. What is the expected value of the Jaccard ...
6
votes
5answers
23k views

Jaccard similarity in R

I want to compare 2 vectors of length 43; they have values of 0 (not present) and 1 (present). I will refer to $M_{1,1}$ as situations in which both 1 are present, and $M_{1,0}$ and $M_{0,1}$ to ...
5
votes
3answers
3k views

Similarity measures for more than 2 variables

If I have two binary variables, I can determine the similarity of these variables quite easily with different similarity measures, e.g. with the Jaccard similarity measure: $J = \frac{M_{11}}{M_{01} +...
4
votes
2answers
2k views

Jaccard similarity coefficient vs. Point-wise mutual information coefficient

Can you explain the difference between the Jaccard similarity coefficient and the pointwise mutual information (PMI) measure? It would be great if you could add a few examples.
4
votes
1answer
1k views

What is the significance of the Jaccard similarity score?

I understand how to calculate the jaccard similarity , but never quite understood the logic behind why are we calculating it. How does it show the similarity between two sets? What relation exactly ...
4
votes
0answers
279 views

Statistical Interpretation of Average Pairwise Similarity

I have assembled binary vectors (0/1 for all elements and equal weight and arranged in time order) that have been separated into different cohorts where a unique event of interest occurs. I have ...
3
votes
1answer
1k views

A similarity measure with binary data: does this one have a name?

There are many binary similarity measures (e.g. Jaccard, Sorensen, etc), each of them is sensitive to different properties of the compared sets. I would like to use the metric $S=\frac{N_{A\bigcap B}}{...
3
votes
2answers
6k views

Accuracy vs Jaccard for multiclass problem

TL;DR For a multiclass problem, is Jaccard score the same as accuracy? Update March 29, 2019 The wrong implementation in scikit-learn is now fixed with pull request #13151. Hooray! P.S. The lesson ...
3
votes
2answers
2k views

Jaccard index between set and multiset

Can I use Jaccard index to calculate similarity between set and multiset? As I know Jaccard is defines as the size of the intersection divided by the size of the union of the sample sets, that is $J(...
3
votes
0answers
269 views

Estimate Jaccard similarity based on a sample

The Jaccard similarity of two sets, $A$ and $B$, is defined as: $Jaccard(A,B)=\frac{A\cap{B}}{A\cup{B}}$. Say that I only have a sample of $P\%$ of each of the sets: $A'$ and $B'$. What would be a ...
3
votes
0answers
107 views

Similarity measures and document length

I have an application where I need to measure the similarity between the (TF-IDF?) representation of two documents: $\mathbf{a}$ and $\mathbf{b}$ while still taking the document length into account. ...
3
votes
1answer
361 views

Is the Jaccard Index a function of the number of elements in a list?

I am trying to understand a research result. Imagine 10 different psychological questionnaires 1,2,3,...10 of different length (i.e. number of questions) that are all supposedly measuring the same ...
3
votes
0answers
466 views

Approach to find similarity between two different types of matrices

I'm using the movielens dataset to give recommendations to a user based on genre of the movies. I have two matrices, one contains the genre the user likes. We are considering user3 for now I also ...
2
votes
1answer
887 views

Apriori versus Jaccard Coeff for recommender-system

I developed a market basket program for a retailer. I used the pymining library association rules function. I assume this uses the Apriori algorithm. I got 340,000 rules for an input of 17,000 ...
2
votes
1answer
2k views

NMDS from Jaccard and Bray-Curtis identical. Is that a bad thing?

The dataset is ecological (species abundance), where I am calculating distances between 51 sampling sites, based on abundances of 3200 species. The species abundance values go up to 2700, and the rest ...
2
votes
2answers
2k views

Significance Test for Jaccard Distance

I am looking for a significance test for the Jaccard Distance (JD). As an example, I have two datasets as follows: Baseline: $\left| A\bigcap B \right|=57;\ \left| A\bigcup B \right|=275\quad \...
2
votes
0answers
1k views

Similarity between sets with different size

is there a distance measure like jaccard for sets with different sizes? For example A=['a','b','c'] and B=['a','d'] I would like to include the total intersection as well as the order. The ...
2
votes
0answers
931 views

Cosine similarity between two types of user profiles

I have a movie dataset containing preference (ratings) of users on movies and also attributes of movies (genre, cast, director). I created two types of profiles for users based on the movie attributes ...
2
votes
0answers
884 views

Finding nearest neighbors using Jaccard distance for positive, real-valued vectors

Say we have $x_i, \ldots, x_n \in R ^ D$ with positive, real components and use Jaccard distance $$d(x_i, x_j) = 1 - \frac{\sum_{d = 1}^D\min(x_i^d, x_j^d)}{\sum_{d = 1}^D\max(x_i^d, x_j^d)}$$ to ...
2
votes
0answers
540 views

A probability distribution model for Jaccard similarity

This is an obfuscated version of a real problem: Each day I speak with some number of (distinct) girls. I compute the Jaccard similarity index between two consecutive days: $$ J(t)=J(t,t-1)=\...
2
votes
1answer
5k views

Jaccard distance vs Levenshtein distance for fuzzy matching

My data is similar to the following data, but far bigger and more complex. Apple Banana Those fruits Tomato Cocumber These vegetables I would like to get the ...
1
vote
1answer
624 views

How to compare a Ranked List?

I have 50 products. For each product, I identify 3 related products (1 top related, 2 (partially related, least related)) using similarity measures. I want to compare the ranked list generated by my ...
1
vote
1answer
263 views

Jaccard Similarity with Binary Data

I have 5400 rows of data and 3211 columns of attributes. The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity ...
1
vote
1answer
35 views

Measuring simmilarity of observations (non numeric)

I have a dataset of format : day,measurement1,measurement2 1,a,b 1,a,c 1,f,s 2,a,b 2,a,c 2,f,g 3,a,d 3,a,q 3,f,s In this example day1 is more similar with day2 ...
1
vote
1answer
414 views

Interpretation of Jaccard similarity for binary networks

Given two undirected networks with the same amount of nodes defined by $A_{ij}^1$ and $A_{ij}^2$, we take the unique unordered pairs $(i, j)$. After sorting them, we then have two binary vectors $\vec{...
1
vote
1answer
1k views

filling out NA values using clustering analysis

I have a data frame with a large number of NA values. I do not wish to leave out all these rows as that would reduce the size of my training set drastically. I filled out these missing values in a ...
1
vote
1answer
949 views

Similarity scoring to compare multi-dimensional datasets

I am trying to come up with a mechanism of scoring a set of multidimensional datasets based on a similarity with an ideal dataset. Each dataset will all have the same dimensions along with the ideal. ...
1
vote
0answers
90 views

Unique Jaccardian Distance?

Hi I have a dataset that could be simulated using: ...
1
vote
0answers
351 views

How do I calculate the Jaccard distance for normal distributions?

I'm trying to replicate the results in this paper. I'm stuck at this part: specifically, calculate Jaccard intersection of pdata1 & pdata2 Is there some neat ...
1
vote
0answers
58 views

Correlation of continuous-time series

I have two continuous time series of discrete events. The data from them are timestamps of occurences, e.g. ...
1
vote
0answers
328 views

Dummy data clustering: OK to apply Euclidian distance hierarchical clustering on Jaccard similarity matrix?

I created dummy variables (binary data) from categorical variables where I want to partition N subjects into multiple classes by some clustering method. I created a Jaccard similarity index matrix for ...
1
vote
0answers
38 views

How to statistically validate that one clustering technique is more effective than other on Binary data

Problem statement - Idea is to cluster students based on their assessment pattern and score. Data = I have a data set that contains Binary data of students answers(Correct/incorrect) which i have ...
1
vote
2answers
269 views

Detecting misclassified data points using distance matrix

I have a data set of around 65,000 items from a public database that have each been assigned to one of around 1,800 classes, but unfortunately I am aware that quite a few are misclassified. My goals ...
1
vote
0answers
142 views

Threshold value for nodes similarity

I am working on an algorithm for community detection on complex systems, where those systems are modeled as graphs. To form community i need to find similarity between nodes. I am using Jaccard ...
1
vote
0answers
220 views

Find similarity score between multiple items based on tags assigned to them

I am working for a client who is in e-commerce. They have thousands of offerings and each has its own set of tags. Now, they want to find similar designs dynamically based on the item selected by the ...
1
vote
1answer
347 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf https://chrisjmccormick.wordpress.com/...
1
vote
0answers
889 views

should I count repeated items in jaccard similarity?

I have two lists: A and B where Size(A) = 4 and size(B) = 10. I would like to find the Jaccard similarity between these two lists. Suppose A = {Tom, George, John, Jennifer} and B={Tom, Jessica, ...
0
votes
3answers
2k views

Can I use Manhattan distance on binary data for hierarchical clustering?

I understand that classically Jaccard and Hamming work best with binary data, but is there anything specifically wrong with using a Manhattan distance instead with the complete linkage function?
0
votes
1answer
919 views

confusion matrix and Jaccard index computation in O(n) for cluster comparison

My problem: I have two clustering (or partitions) of a set of numbers $\{1,...,n\}$, each with $k$ cluster (or subsets), that i have to compare and, to do that, i'd like to try some different indexes ...
0
votes
2answers
1k views

Matrix clustering based on a Jaccard distance cutoff

I'm trying to figure out how to group elements of a binary matrix based on a given Jaccard distance cutoff. For example, suppose that I have information on the types of food carried by various grocery ...
0
votes
1answer
44 views

distance metric for student course schedules

I'm doing an exploratory clustering analysis of student course schedules at a college. Interpretability by humans is paramount: we're trying to inform future research questions and possibly ...
0
votes
2answers
649 views

Jaccard similarity of sets with approximate element matching

Consider two sets like this: A = {'hello', 'world'} and B = {'Hello', 'helo', 'goodbye'}. I want to measure the similarity between these two sets. There are two levels to this problem: Comparing ...
0
votes
1answer
94 views

How to soften or mitigate vector similarity measure?

I would like to evaluate a similarity between two objects X and Y by comparing a neighbourhood in which they're located. I construct two sets of nine concentric and equidistant circles with centers ...
0
votes
1answer
290 views

Is there a distance algorithm similar to Jaccard distance that handles scalar data?

we have the characteristics and (scalar) values of those characteristics for three (or more) people: ...
0
votes
1answer
26 views

How to create a “similarity index” on a set of variables

I have a set of 12 variables that describe the dwelling characteristics. 10 of them are dummies (i.e questions like "does the dwelling have a bath?" or "is there crime in the area?"), while one is a 4-...
0
votes
0answers
17 views

Jaccard Simialrity between different populations

I want to calculate Jaccard similarity (in R) between ten populations from two types of streams(contaminated and non-contaminated), five from each. I want to compare the similarity between these ...
0
votes
0answers
24 views

Set Similarity Measure that accounts for size

Suppose I have two sets, A and B where 10|A| = |B| and A is a subset of B. For the case where |A| = 1 the Jaccard similarity of the sets will be 0.1. For the case where |A| = 100 the Jaccard ...
0
votes
1answer
63 views

Similarity between n >> 2 binary vectors

I have $n$ binary vectors $x_i, ..., x_n$ of length $d$. I wish to compute a similarity measure between all $n$ of them. My initial thought was to use the Jaccard Index, since each binary element $x_{...
0
votes
0answers
321 views

Jaccard Index for Binary Data in R using dist function

I have a presence/absence table of 0s and 1s and I would like to cluster this data. I want to create a pairwise matrix using R's dist function which has a binary ...