I have a dataset with each row a country and 10 columns with numerical features like GDP,Electrcity consumption, GNI etc. I am trying to use distance metrics to find similarity between the countries and ultimately cluster them. I have tried quite a few distance metrics like Euclidean, Minkowski, canberra, jaccard etc. In case of jaccard (implementation in pdist in scipy) I don't think the resulting dissimilarity matrix makes sense as I have all 1's in the matrix other than 0 along diagonal. I read more on jaccard and it seems to use set union and intersection in the computation. So am I wrong to apply it in case of continous variables? I have a read a lot on jaccard and it seems to be useful only when data is represented in terms of 0/1 (present/absent). Please guide :)
-
1$\begingroup$ The tl;dr answer is: Don't use it for continuous data. You are very right, it doesn't make sense. $\endgroup$– Christian HennigCommented Aug 20, 2020 at 15:30
2 Answers
Originally, Jaccard similarity is defined on binary data only. However, its idea (as correctly displayed by @ping in their answer) could be attempted to extend over to quantitative (scale) data. In many sources, Ruzicka similarity is being seen as such equivalent of Jaccard. A screenshot from the document of my SPSS macro !PROXQNT (can be found on my web-page, "Various proximities" collection):
Besides this, one should also keep in mind that in case of binary data, Jaccard sim = Ruzicka sim (= 1 - Soergel dis) = Similarity ratio = Ellenberg sim.
Therefore per backward logic, Similarity ratio and Ellenberg similarity can be considered too, as other candidates for the equivalence towards Jaccard.
-
$\begingroup$ Hi thank you so much for this detailed explanation. I tried to use the soergel distance formula in python to calculate pairwise distances for my matrix. However the distances are not in the range of 0,1. I read somewhere that Soergel distance must be between 0 and 1. Is this correct? Thanks $\endgroup$– skynaiveCommented Aug 21, 2020 at 9:19
-
1$\begingroup$ From the formula, clearly, it cannot exceed 1. The data must be nonnegative. $\endgroup$– ttnphnsCommented Aug 21, 2020 at 10:04
-
$\begingroup$ I checked an implementation of soergel distance in R, it also won't give a value b/w 0 and 1. So now I am confused. The input are 2 numeric vectors x and y. [link]rdrr.io/cran/philentropy/man/soergel.html I just realised my data has some negative values due to scaling. Is this alright? Or is there a strict need for non negative values for soergel? $\endgroup$– skynaiveCommented Aug 21, 2020 at 11:09
-
1$\begingroup$ I think it won't make sense with negative values. Similarity ratio however is suitable with data of any sign. Similarity ratio has some kinship with correlation coef., see stats.stackexchange.com/a/22520/3277 $\endgroup$– ttnphnsCommented Aug 21, 2020 at 11:33
-
$\begingroup$ Thank you. I didnt know that some metrics don't make sense if we have negative values in dataset. As i used manhattan,euclidean,canberra etc on this data. Is there any good resource which lists what kind of data suits what distance metric. I have tried to look for this however most just give a formula and not much else. Thanks $\endgroup$– skynaiveCommented Aug 21, 2020 at 11:40
Jaccard similarity is, in general, valid for any pair of sets https://en.wikipedia.org/wiki/Jaccard_index
Given two sets $A$ and $B$:
$$ J=\frac{|A \cap B|}{|A \cup B|} $$
No requirement is given about the elements of $A$ and $B$. In general, it can be seen as the relative (Lebesgue) measure, between the intersection and the union of the two sets. Under this interpretation, it may be applied to all pairs of elements of a measurable space $X$. When $X$ is a Borel space, with $\sigma$-algebra $\Sigma$, the measure can be also probabilistic:
$$ J=\frac{\mu(A \cap B)}{\mu(A \cup B)} $$
given that $A, B \in \Sigma$.
-
1$\begingroup$ Sorry I am not clear with this answer. So sets A is 1 row in my ds like 'USA':{0.23,1.3,5.67,1.06} set B is another row 'AR':{1.33,0.4,0.67,5.5} here the set element sare continous features like GDP,GNI, etc as said in my question above. So when we calculate union of A and B or intersection. There will never be any elements in common as these are continous features. So how does jaccard apply in my case. $\endgroup$– skynaiveCommented Aug 20, 2020 at 14:08
-
1$\begingroup$ The problem is that the Jaccard doesn’t capture the similarity you are trying to estimate. As you asked for the applicability of the Jaccard, the answer is yes, it can be applied to any pair of sets. $\endgroup$ Commented Aug 20, 2020 at 14:25
-
$\begingroup$ You need another similar measure. $\endgroup$ Commented Aug 9, 2022 at 3:38