I have three judges (A,B,C) that estimate a value for N objects. Each judge estimates the same object three times, so that my data frame looks like
object A1 A2 A3 B1 B2 B3 C1 C2 C3 1 19.93 20.05 20.15 20.01 20.66 19.72 20.53 21.02 20.41 2 19.78 19.16 19.47 19.90 20.50 19.60 20.41 20.76 20.42 3 19.22 19.33 19.41 19.82 20.39 19.49 20.33 20.69 20.32 4 19.21 19.67 19.41 19.76 20.34 19.42 20.30 20.67 20.29
Now I would like to test whether the variation beetween judges is greater than within each judge. The problem is, that each object is different, which means that the rows connot be considered measurements of the same random varaible, but the true value differs from object to object.
Does someone know of a simple test for this problem?
Here is what came to my mind:
Compare the average spans (max - min) per object within each judge with the span between all judges. From a practical point of view, the absolute value of thsi difference is is more important than a significance test, but it would nevertheless be interesting to have a test for statistical significance.
Do a paired t-test for all combinations (AB, AC, BC) with a Bonferroni-correction. This raises the question, how the three observations of each judge are taken into account. By adding differences of all combinations?
Use ICC (intra-class correlation) for the average values of each judge. This approach, however, throws away two third of the data, because it reduces the three values per judge per object to a single one.