# How to compare annotations by two groups?

I have two groups of test persons that manually created multiple annotations for different underlying, unknown values (each person created multiple annotations). I'd like to find a statistical test that tells me with a given confidence level that the annotations by the two groups are of the same quality. Quality being expressed in terms of the variance of their annotations.

(Artificial) examples for annotations:

Annotation by person1: {100, 101, 99, 98, 100, 101}
Annotation by person1: {50, 49, 49, 49, 50, 51}
Annotation by person2: {12, 12, 13, 12}
...


To compare the series individually, one can use the Coefficient of Variation, i.e. $c_v=\frac{\sigma}{\mu}$. This addresses the fact that the series all have different means and describe different true values. And to compare two $c_v$ one can use the Asymptotic test for the equality of coefficients of variation from k populations (Feltz and Miller 1996).

But how do I test for equality of many (hundreds) time series (each with a different mean, but similar $c_v$) from two groups? What I really want to test is the equality of the annotations by group.

What if I simply compute the $c_v$ for all series and treat these $c_v$-values as two distributions. One for group1 and one for group2. I could test for significant differences in these two distributions using t test. Would this approach allow me to judge with confidence whether the annotations by members of the two groups are of equal quality?

If not, can you suggest a better approach?

Thank you.

• Are the annotators working on the same work items? – Vladislavs Dovgalecs Feb 16 '18 at 17:26
• Some items may be identical, others won't. For each item, an annotator will produce multiple annotations, as shown in the example. – hendrik Feb 16 '18 at 17:28
• You may want to read this paper aclweb.org/anthology/Q14-1025 on the probabilistic approach. There are other models that estimate worker reliability and item difficulty (difficulty-ability models). I have some simple implementations in C#. This requires some familiarity with Bayesian statistics, graphical models in particular. – Vladislavs Dovgalecs Feb 16 '18 at 17:37
• Thank you. What about the much simpler t-test approach I outlined. Will that produce respectable results? – hendrik Feb 18 '18 at 9:48