2
$\begingroup$

I'm looking for an appropriate test to assess significance of (dis)similarity between binary vectors. Working example in R: denote the values of 10 features of 5 products by 0 (no feature present) or 1 (product has this feature)

mtrx <- rbind(c(rep(1, 4), rep(0, 6)),
      c(0, rep(1, 4), rep(0, 5)),
      c(rep(0, 3), rep(1, 3), rep(0, 4)),
      c(rep(0, 6), rep(1, 4)),
      c(rep(0, 6), rep(1, 3), 0))

In my actual dataset the number of features is of the order of thousands and the number of 1s per product varies wildly. I'd like to (i) have a measure of assessing (dis)similarity of products and (ii) assessing how significant this is.

Many measures came to my mind: Hamming distance, MCC etc. For instance,

sum(mtrx[1, ] != mtrx[2, ]) # Hamming distance of products 1 and 2 is 2

To assess whether this is significant or not I tried permutation tests, for instance (null hypothesis is that they are the same)

sim <- replicate(10000, sample(c(mtrx[1, ], mtrx[2, ])))
sum(colSums(sim[1:10, ] != sim[11:20, ]) >= 2 )/ 10000

In my real dataset almost all "p-values" I obtain this way are zero, since it's impossible to create all possible permutations. How could I add a confidence interval to the calculated p-value? Is there a better way to do the whole thing?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.