I perform a experiment on which the measured results are codes into two variables: $y_1 \in {-1,0,1}$ and $y_2 \in {-1,0,1}$.

While $-1$ and $1$ are opposite result, $0$ is coding a bad (uncertain) measurement, and is very unlikely, so in fact both of $y_1, y_2$ are almost binary.

How do I check the correlation (or other matching) between $y_1$ and $y_2$?

For example, Pearson is quiet bad in describing this phenomena, since in case of following counts:



It shows a bad correlation ($R^2= 0.196$) when in fact, there is a good match (9/12) between the cross-counts of the two.

On the other side, if I use some binary similarity measure such as Hamming similarity, I'll get a $0.75$ similarity (only 3/12 mismatches), but in this measure I'm missing the significance (what is the P-value of matching?).

Any help in choosing the right measure of significant matching?

  • $\begingroup$ What would you imagine the null hypothesis could be? Does the order matter as you suggest in your other question? $\endgroup$
    – vkehayas
    Sep 4 '17 at 19:28
  • 1
    $\begingroup$ @Dimgold As far as correlation between ordinal variables is concerned, Spearman's rho/ Kendall rank are better suited than Pearson. $\endgroup$
    – Ketan
    Sep 4 '17 at 20:00
  • $\begingroup$ If you want, you can shift the distribution from y \in -1,0,1 to y \in 0, 1, 2 and do a kullback-leibler divergence test. When I did this I got 0.0437, meaning they're pretty similar. $\endgroup$
    – Greg
    Sep 4 '17 at 20:51
  • $\begingroup$ @vkehayas - The null hypothesis is that the variables are unrelated (And I'm looking for the right mathematical definition) $\endgroup$
    – Dimgold
    Sep 6 '17 at 7:20

You can actually compute the p-value of any similarity you like, in a similar fashion as the p-value for the Pearson correlation is computed.

The simplest way is to do a permutation test or a bootstrap. A good description can be found here.

Permutation test

In the permutation test, you can:

1) randomly shuffle your two vectors and compute the Hamming similarity.

2) repeat (1) many times (say N = 1e6) and measure the number of times X that the random similarity is larger than the value you observed (0.75).

Your p-value (one-sided) will simply be the number X/N (the proportion of times the random vectors were more similar than the original vectors).


As an alternative you can use bootstrapping. The difference is in step (1). Instead of shuffling the vectors, you sample their values with replacement. For example, to randomise this vector:

y1 =[−1,−1,−1,−1,−1,−1,−1,−1,0,−1,1,1]

you produce a vector of 12 entries sampled with probabilities:

p(0) = 1/12

p(1) = 2/12

p(-1) = 9/12

This will preserve the proportions of 1,0 and -1 on average, while the permutation test will preserve that exactly (that's why that is also called exact test).

If you want to test another null hypothesis, you can also sample the values with uniform probability. That will likely give you smaller p-values.

  • $\begingroup$ Is there a formal name to this method? Can I report bootstapping as a formal result in a academic paper? $\endgroup$
    – Dimgold
    Sep 6 '17 at 7:21
  • 1
    $\begingroup$ The method I proposed is called "permutation test". It falls under the hood of "resampling" methods. So, yes you can report it as a permutation test (it does not hurt to explain what the test is exactly in the supplementary material). Bootstrapping is another resampling method, but it's "with replacement", so it's slightly different. $\endgroup$
    – AndreaL
    Sep 6 '17 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.