I have two groups of test persons that manually created multiple annotations for different underlying, unknown values (each person created multiple annotations). I'd like to find a statistical test that tells me with a given confidence level that the annotations by the two groups are of the same quality. Quality being expressed in terms of the variance of their annotations.

(Artificial) examples for annotations:

Annotation by person1: {100, 101, 99, 98, 100, 101}
Annotation by person1: {50, 49, 49, 49, 50, 51}
Annotation by person2: {12, 12, 13, 12}

To compare the series individually, one can use the Coefficient of Variation, i.e. $c_v=\frac{\sigma}{\mu}$. This addresses the fact that the series all have different means and describe different true values. And to compare two $c_v$ one can use the Asymptotic test for the equality of coefficients of variation from k populations (Feltz and Miller 1996).

But how do I test for equality of many (hundreds) time series (each with a different mean, but similar $c_v$) from two groups? What I really want to test is the equality of the annotations by group.

What if I simply compute the $c_v$ for all series and treat these $c_v$-values as two distributions. One for group1 and one for group2. I could test for significant differences in these two distributions using t test. Would this approach allow me to judge with confidence whether the annotations by members of the two groups are of equal quality?

If not, can you suggest a better approach?

Thank you.

  • $\begingroup$ Are the annotators working on the same work items? $\endgroup$ – Vladislavs Dovgalecs Feb 16 '18 at 17:26
  • $\begingroup$ Some items may be identical, others won't. For each item, an annotator will produce multiple annotations, as shown in the example. $\endgroup$ – hendrik Feb 16 '18 at 17:28
  • $\begingroup$ You may want to read this paper aclweb.org/anthology/Q14-1025 on the probabilistic approach. There are other models that estimate worker reliability and item difficulty (difficulty-ability models). I have some simple implementations in C#. This requires some familiarity with Bayesian statistics, graphical models in particular. $\endgroup$ – Vladislavs Dovgalecs Feb 16 '18 at 17:37
  • $\begingroup$ Thank you. What about the much simpler t-test approach I outlined. Will that produce respectable results? $\endgroup$ – hendrik Feb 18 '18 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.