Statistical significance for many instances of Kendall tau correlation

Question

I am working on a dataset, which contains multiple instances of ranking vectors, done by two annotators.

In particular, I have about 20,000 data samples. Each sample contains a ranking of 4-5 items done twice: Once by a human and once by a statistical model.

Example:

Human [1 2 3 4] [1 3 2 5 4] ... [2 3 4 1 5]
Model [2 3 4 1] [1 2 3 5 4] ... [1 2 3 5 4] (plus 19,997 more samples like this)

So far I have used Kendall's tau in order to measure the correlation between the human and the statistical model, by summing all discordant and concordant rank pairs of all samples. But I am having difficulties to find the proper way to calculate statistical significance for the null hypothesis test, that the two sets of ranking vectors each are independent.

Having read the original Kendall's theory, there is a formula for calculating the p-value, given that you have one pair of wannabee correlated ranking vectors with more than 10 ranks each. On the contrary, I want to do a significance test when I have many pairs of ranking vectors, each of them at most 5 ranks long.

Additionally, there is also some analysis for the significance test when having more than two annotators, that they all annotate the same sample. But this is again not what I want.

Answer 1

If you're interested in whether the average correlation is higher than zero, this should be straightforward

Under a null of no correlation, the individual Kendall correlations of $m$ elements should have mean 0 and variance $\frac{2(2m+5)}{9m (m-1)}\,$. This variance formula applies down to $m=2$.

Imagine there are $n_k$ cases where $m=k$.

Then the sum of $n_k$ independent such correlations $\sum_{i=1}^{n_k} \hat{\tau}_i$ has mean 0 and variance $n_k\frac{2(2m+5)}{9m (m-1)}$, and the average at $m=k$, $\bar{\tau}^{(k)}=\frac{1}{n_k}\sum_{i=1}^{n_k} \hat{\tau}^{(k)}_i$ then has mean 0 and variance $\frac{1}{n_k}\frac{2(2m+5)}{9m (m-1)}$.

The estimate of $\tau$ with smallest variance would weight inversely proportional to variance. That is $\hat{\tau}=\sum_k w_k \bar{\tau}^{(k)}/\sum_k w_k$, where $w_k=n_k\frac{9m (m-1)}{2(2m+5)}$.

The variance of this can be computed by the basic properties of variance.

We have that $\overline{\tau}$ should be approximately standard normal under the null. You could then either set up a one- or two-tailed test, as you wished. (A continuity correction would improve the approximation but with large $n$ should not be necessary.)

[If you want to consider something other than the average correlation as a statistic there'd be a variety of other ways of combining the individual information into a single overall statistic, but it depends on what you're interested in.]

However, with an $n$ of 20000, you might want to consider that you're almost certain to reject the null at any reasonable significance level. It might make more sense to try to construct a confidence interval for the Kendall correlation.