I have 8 conditions. In a pairwise experiment, I am comparing 28 pairs: condition1 vs condition2, condition3 vs. condition 4, condition1 vs condition3, condition4 vs, condition2, etc etc.
Each participant in my experiment sees 10 randomly sampled pairs (so they don't see all 28 pairs). I am attempting to compute kendalls u coefficient to calculate overall consistency (see overall consistency in this paper: http://mmnet.iis.sinica.edu.tw/pub/chen09_crowdsourcing.pdf) but I can't since every participant did not see every pair. Is there an alternate algorithm to calculate overall consistency that is used in academic papers?