2
$\begingroup$

I have a dataset with each row a country and 10 columns with numerical features like GDP,Electrcity consumption, GNI etc. I am trying to use distance metrics to find similarity between the countries and ultimately cluster them. I have tried quite a few distance metrics like Euclidean, Minkowski, canberra, jaccard etc. In case of jaccard (implementation in pdist in scipy) I don't think the resulting dissimilarity matrix makes sense as I have all 1's in the matrix other than 0 along diagonal. I read more on jaccard and it seems to use set union and intersection in the computation. So am I wrong to apply it in case of continous variables? I have a read a lot on jaccard and it seems to be useful only when data is represented in terms of 0/1 (present/absent). Please guide :)

$\endgroup$
1
  • 1
    $\begingroup$ The tl;dr answer is: Don't use it for continuous data. You are very right, it doesn't make sense. $\endgroup$ – Lewian Aug 20 '20 at 15:30
1
$\begingroup$

Originally, Jaccard similarity is defined on binary data only. However, its idea (as correctly displayed by @ping in their answer) could be attempted to extend over to quantitative (scale) data. In many sources, Ruzicka similarity is being seen as such equivalent of Jaccard. A screenshot from the document of my SPSS macro !PROXQNT (can be found on my web-page, "Various proximities" collection):

enter image description here

Besides this, one should also keep in mind that in case of binary data, Jaccard sim = Ruzicka sim (= 1 - Soergel dis) = Similarity ratio = Ellenberg sim.

Therefore per backward logic, Similarity ratio and Ellenberg similarity can be considered too, as other candidates for the equivalence towards Jaccard.

enter image description here

$\endgroup$
5
  • $\begingroup$ Hi thank you so much for this detailed explanation. I tried to use the soergel distance formula in python to calculate pairwise distances for my matrix. However the distances are not in the range of 0,1. I read somewhere that Soergel distance must be between 0 and 1. Is this correct? Thanks $\endgroup$ – skynaive Aug 21 '20 at 9:19
  • 1
    $\begingroup$ From the formula, clearly, it cannot exceed 1. The data must be nonnegative. $\endgroup$ – ttnphns Aug 21 '20 at 10:04
  • $\begingroup$ I checked an implementation of soergel distance in R, it also won't give a value b/w 0 and 1. So now I am confused. The input are 2 numeric vectors x and y. [link]rdrr.io/cran/philentropy/man/soergel.html I just realised my data has some negative values due to scaling. Is this alright? Or is there a strict need for non negative values for soergel? $\endgroup$ – skynaive Aug 21 '20 at 11:09
  • 1
    $\begingroup$ I think it won't make sense with negative values. Similarity ratio however is suitable with data of any sign. Similarity ratio has some kinship with correlation coef., see stats.stackexchange.com/a/22520/3277 $\endgroup$ – ttnphns Aug 21 '20 at 11:33
  • $\begingroup$ Thank you. I didnt know that some metrics don't make sense if we have negative values in dataset. As i used manhattan,euclidean,canberra etc on this data. Is there any good resource which lists what kind of data suits what distance metric. I have tried to look for this however most just give a formula and not much else. Thanks $\endgroup$ – skynaive Aug 21 '20 at 11:40
1
$\begingroup$

Jaccard similarity is, in general, valid for any pair of sets https://en.wikipedia.org/wiki/Jaccard_index

Given two sets $A$ and $B$:

$$ J=\frac{|A \cap B|}{|A \cup B|} $$

No requirement is given about the elements of $A$ and $B$. In general, it can be seen as the relative (Lebesgue) measure, between the intersection and the union of the two sets. Under this interpretation, it may be applied to all pairs of elements of a measurable space $X$. When $X$ is a Borel space, with $\sigma$-algebra $\Sigma$, the measure can be also probabilistic:

$$ J=\frac{\mu(A \cap B)}{\mu(A \cup B)} $$

given that $A, B \in \Sigma$.

$\endgroup$
2
  • $\begingroup$ Sorry I am not clear with this answer. So sets A is 1 row in my ds like 'USA':{0.23,1.3,5.67,1.06} set B is another row 'AR':{1.33,0.4,0.67,5.5} here the set element sare continous features like GDP,GNI, etc as said in my question above. So when we calculate union of A and B or intersection. There will never be any elements in common as these are continous features. So how does jaccard apply in my case. $\endgroup$ – skynaive Aug 20 '20 at 14:08
  • $\begingroup$ The problem is that the Jaccard doesn’t capture the similarity you are trying to estimate. As you asked for the applicability of the Jaccard, the answer is yes, it can be applied to any pair of sets. $\endgroup$ – osmoc Aug 20 '20 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.