In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of the dataset, there is concern whether the dataset can still be used. What is your opinion?

Some more details:

Let there be a gold standard dataset D1 created by averaging ratings of 13 annotators. The ratings are on scale 0 to 10. The interannnotator agreement is 0.6 as computed as the average of pairwise Spearman correlations between the ratings of all raters.

Say the state-of-the-art algorithm obtains Spearman correlation of 0.8 with the gold standard. Since this exceeds the interannotator agreement of 0.6, the algorithm is better than the humans. Can the gold standard still be used to track progress in algorithm development in the given field, or is a new dataset with higher interannotator agreement needed?

I would especially appreciate any references to literature dealing with this topic.

  • $\begingroup$ Just out of curiosity: How does one obtain different annotations from a single algorithm to compute the pairwise correlations? $\endgroup$ – ziggystar Aug 7 '15 at 9:41
  • $\begingroup$ Did you get any updates on this, tomas? What are you current thoughts? $\endgroup$ – michal Sep 20 '17 at 9:37
  • $\begingroup$ Unfortunately no. I have not made any progress myself on this. $\endgroup$ – tomas Sep 23 '17 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.