I am working on a dataset, which contains multiple instances of ranking vectors, done by two annotators.

In particular, I have about 20,000 data samples. Each sample contains a ranking of 4-5 items done twice: Once by a human and once by a statistical model.


Human [1 2 3 4] [1 3 2 5 4] ... [2 3 4 1 5]
Model [2 3 4 1] [1 2 3 5 4] ... [1 2 3 5 4] (plus 19,997 more samples like this)

So far I have used Kendall's tau in order to measure the correlation between the human and the statistical model, by summing all discordant and concordant rank pairs of all samples. But I am having difficulties to find the proper way to calculate statistical significance for the null hypothesis test, that the two sets of ranking vectors each are independent.

Having read the original Kendall's theory, there is a formula for calculating the p-value, given that you have one pair of wannabee correlated ranking vectors with more than 10 ranks each. On the contrary, I want to do a significance test when I have many pairs of ranking vectors, each of them at most 5 ranks long.

Additionally, there is also some analysis for the significance test when having more than two annotators, that they all annotate the same sample. But this is again not what I want.


If you're interested in whether the average correlation is higher than zero, this should be straightforward

Under a null of no correlation, the individual Kendall correlations of $m$ elements should have mean 0 and variance $\frac{2(2m+5)}{9m (m-1)}\,$. This variance formula applies down to $m=2$.

Imagine there are $n_k$ cases where $m=k$.

Then the sum of $n_k$ independent such correlations $\sum_{i=1}^{n_k} \hat{\tau}_i$ has mean 0 and variance $n_k\frac{2(2m+5)}{9m (m-1)}$, and the average at $m=k$, $\bar{\tau}^{(k)}=\frac{1}{n_k}\sum_{i=1}^{n_k} \hat{\tau}^{(k)}_i$ then has mean 0 and variance $\frac{1}{n_k}\frac{2(2m+5)}{9m (m-1)}$.

The estimate of $\tau$ with smallest variance would weight inversely proportional to variance. That is $\hat{\tau}=\sum_k w_k \bar{\tau}^{(k)}/\sum_k w_k$, where $w_k=n_k\frac{9m (m-1)}{2(2m+5)}$.

The variance of this can be computed by the basic properties of variance.

We have that $\overline{\tau}$ should be approximately standard normal under the null. You could then either set up a one- or two-tailed test, as you wished. (A continuity correction would improve the approximation but with large $n$ should not be necessary.)

[If you want to consider something other than the average correlation as a statistic there'd be a variety of other ways of combining the individual information into a single overall statistic, but it depends on what you're interested in.]

However, with an $n$ of 20000, you might want to consider that you're almost certain to reject the null at any reasonable significance level. It might make more sense to try to construct a confidence interval for the Kendall correlation.

  • $\begingroup$ Thanks for the useful answer. I'd need some clarifications: 1. According to Kendall, the variance is given this way if m>10. Can we also use it when we have many samples but still short ranks (m=5)? 2. In my data I have 5-rank, 4-rank etc samples, so the variance of individual tau correlations will vary. Should I calculate the actual sum of variances based on the real data? 3. Since sample length varies, there is a difference between micro-averaged and macro-avgd tau. Does this change anything for the hypothesis test? 4. Can I have a hint for the confidence interval? $\endgroup$ – lefterav May 10 '15 at 0:35
  • 1
    $\begingroup$ 1. (i) I believe you're misinterpreting. Where does Kendall say that the variance formula only applies for m>10? (ii) The variance under the null at m=5 is definitely 1/6; I double-checked by complete enumeration of the permutation distribution (easy in R). 2. Sorry, I missed that they weren't all 5. Is there a known largest and smallest value? Do you know how many of each $m$ there are? 3. I've written a partial explanation that takes account of it; when I know the answers to the previous questions I could try to say more. 4. A confidence interval for the Kendall correlation? I'll have a go $\endgroup$ – Glen_b May 10 '15 at 1:53
  • $\begingroup$ 1. yes you are right 2. their values are between 2 and 12. I don't have the exact measures, but your formulas are pretty clear. 3 I mean that averaging taus calculated from all ranks of different length, has a different value than calculating tau from the overall some of dis/con-cordant pairs of all ranks. $\endgroup$ – lefterav May 13 '15 at 0:03
  • 1
    $\begingroup$ In respect of (1), what would change at somewhere around 10 would be the usefulness of the normal approximation. That won't be a consideration here. In respect of (2) it would simply offer some simplification of the presentation, so if you follow okay there's no need to pursue that. In respect of your 3. there, I'm not quite sure what you're doing in the second part, in particular with what the ranking is doing; perhaps you could expand on exactly what is being calculated there in the question (where there's room). I realize I haven't tackled the CI yet. I'll see if I can look at it today. $\endgroup$ – Glen_b May 13 '15 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.