I'm looking for an appropriate test to assess significance of (dis)similarity between binary vectors. Working example in R: denote the values of 10 features of 5 products by 0 (no feature present) or 1 (product has this feature)

mtrx <- rbind(c(rep(1, 4), rep(0, 6)),
      c(0, rep(1, 4), rep(0, 5)),
      c(rep(0, 3), rep(1, 3), rep(0, 4)),
      c(rep(0, 6), rep(1, 4)),
      c(rep(0, 6), rep(1, 3), 0))

In my actual dataset the number of features is of the order of thousands and the number of 1s per product varies wildly. I'd like to (i) have a measure of assessing (dis)similarity of products and (ii) assessing how significant this is.

Many measures came to my mind: Hamming distance, MCC etc. For instance,

sum(mtrx[1, ] != mtrx[2, ]) # Hamming distance of products 1 and 2 is 2

To assess whether this is significant or not I tried permutation tests, for instance (null hypothesis is that they are the same)

sim <- replicate(10000, sample(c(mtrx[1, ], mtrx[2, ])))
sum(colSums(sim[1:10, ] != sim[11:20, ]) >= 2 )/ 10000

In my real dataset almost all "p-values" I obtain this way are zero, since it's impossible to create all possible permutations. How could I add a confidence interval to the calculated p-value? Is there a better way to do the whole thing?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.