I am working on a dataset, which contains multiple instances of ranking vectors, done by two annotators.
In particular, I have about 20,000 data samples. Each sample contains a ranking of 4-5 items done twice: Once by a human and once by a statistical model.
Human [1 2 3 4] [1 3 2 5 4] ... [2 3 4 1 5] Model [2 3 4 1] [1 2 3 5 4] ... [1 2 3 5 4] (plus 19,997 more samples like this)
So far I have used Kendall's tau in order to measure the correlation between the human and the statistical model, by summing all discordant and concordant rank pairs of all samples. But I am having difficulties to find the proper way to calculate statistical significance for the null hypothesis test, that the two sets of ranking vectors each are independent.
Having read the original Kendall's theory, there is a formula for calculating the p-value, given that you have one pair of wannabee correlated ranking vectors with more than 10 ranks each. On the contrary, I want to do a significance test when I have many pairs of ranking vectors, each of them at most 5 ranks long.
Additionally, there is also some analysis for the significance test when having more than two annotators, that they all annotate the same sample. But this is again not what I want.