Significance Test for Jaccard Distance

Question

I am looking for a significance test for the Jaccard Distance (JD).

As an example, I have two datasets as follows:

Baseline: $\left| A\bigcap B \right|=57;\ \left| A\bigcup B \right|=275\quad \therefore \ JD=0.7927$

Evaluation: $\left| A\bigcap B \right|=126;\ \left| A\bigcup B \right|=433\quad \therefore \ JD=0.7090$

Is there a way to determine whether the JD at evaluation is significantly different from the baseline?

Or do I simply use the classical z-test of proportions? The z-test assumes that there is a significant difference from the baseline.

Answer 1

I found an article that describes the Jaccard index from a probabilistic perspective written by Real and Vergas in 1996: The Probabilistic Basis of Jaccard's Index of Similarity. A few years later, they even published tables of significance values (Table 3) in: Tables of significant values of Jaccard's index of similarity. Though they describe how to determine if J is significant, it may not directly answer your question... However, the statistical appoach given in (Real and Vergas, 1996) may be helpful to derive an appropriate methodology.

Btw, I would recommend not to use a z-test as I sometimes obtain significant results even for small differences between means due to small standard deviations... So to me, this test seems to be a bit overoptimitic and should not be applied when using (in my case) cross-validation/bootstrapping or similar approaches to assess the stability of estimates...

Answer 2

The Jaccard distance/index/coefficient (also known as the Tanimoto index/coefficient) is a popular measure for similarity/dissimilarity between binary data. We can directly compute the statistical significance of the Jaccard index/coefficient using a R package, jaccard on CRAN

Please note that the Jaccard distance is complementary to the Jaccard coefficient Wikipedia. So you can subtract the Jaccard coefficient from 1 to get the Jaccard distance.

Given two input vectors, its main function, jaccard.test(), computes a p-value. If your data is too big, the exact solution (accessed through method = "exact") could be slow and you may want to use a fast and accurate estimation (access through method = "mca").