In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of the dataset, there is concern whether the dataset can still be used. What is your opinion?
Some more details:
Let there be a gold standard dataset D1 created by averaging ratings of 13 annotators. The ratings are on scale 0 to 10. The interannnotator agreement is 0.6 as computed as the average of pairwise Spearman correlations between the ratings of all raters.
Say the state-of-the-art algorithm obtains Spearman correlation of 0.8 with the gold standard. Since this exceeds the interannotator agreement of 0.6, the algorithm is better than the humans. Can the gold standard still be used to track progress in algorithm development in the given field, or is a new dataset with higher interannotator agreement needed?
I would especially appreciate any references to literature dealing with this topic.