I'm looking for an appropriate test to assess significance of (dis)similarity between binary vectors. Working example in R: denote the values of 10 features of 5 products by 0 (no feature present) or 1 (product has this feature)
mtrx <- rbind(c(rep(1, 4), rep(0, 6)), c(0, rep(1, 4), rep(0, 5)), c(rep(0, 3), rep(1, 3), rep(0, 4)), c(rep(0, 6), rep(1, 4)), c(rep(0, 6), rep(1, 3), 0))
In my actual dataset the number of features is of the order of thousands and the number of 1s per product varies wildly. I'd like to (i) have a measure of assessing (dis)similarity of products and (ii) assessing how significant this is.
Many measures came to my mind: Hamming distance, MCC etc. For instance,
sum(mtrx[1, ] != mtrx[2, ]) # Hamming distance of products 1 and 2 is 2
To assess whether this is significant or not I tried permutation tests, for instance (null hypothesis is that they are the same)
sim <- replicate(10000, sample(c(mtrx[1, ], mtrx[2, ]))) sum(colSums(sim[1:10, ] != sim[11:20, ]) >= 2 )/ 10000
In my real dataset almost all "p-values" I obtain this way are zero, since it's impossible to create all possible permutations. How could I add a confidence interval to the calculated p-value? Is there a better way to do the whole thing?