How many annotators are required for creation of a reliable ground truth for a music emotion recognition task? I am doing a research about music emotion recognition. In my research, I use a two-dimensional (Valence and Arousal) space to represent the emotion of a song. To validate my approach, I have to create a ground truth of music emotion. Most of literature I have read hired about 5 people to annotate the emotion of a song from the list of songs in question. I wonder that whether there is a study which examined or specified the number of annotators for a music emotion recognition task? I have searched for that study but I can not find it. Thank you for all your answers!
 A: There are two questions when choosing the number of annotators:


*

*How accurate are they

*How much they disagree


For example, if all annotator give similar answers, one will be enough.
In most real world situations, most human annotator are quite accurate (but not prefect) and tend to agree with each other but not always.
In such cases you can aggregate the results of the different annotators in order to improve your labeling. A classical way to do so is to use a Dawid-Skene estimator (Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm).
In this method we use the performance of each annotator. Note that using a majority vote might not be good enough since you might have a very accurate annotator disagreeing with many bad ones. Given the performance you estimate the labels and then re estimate the annotator. This technique is called Expectation maximization.
Please note that in many cases, using this technique will be an over kill. Instead, estimate the performance of each annotator with respect to some gold standard. After that, give the majority rule a try. If the majority rule results satisfy you, you are good to go. You might even find out the 3 annotators will be enough.
A: If still interested in this, I recently taught a class at MIT on precisely this topic: https://dcai.csail.mit.edu/lectures/dataset-creation-curation/
It covers various methods to establish consensus labels from many noisy annotations (eg. from crowdsourced data labelers) and to estimate our confidence that each consensus label is actually correct. And also how to estimate the amount of data we'll need to label.
Methods better than Dawid-Skene (like the CROWDLAB algorithm described in the linked lecture notes)  train a ML classifier to augment the data annotators when estimating consensus (and confidence therein). Good estimates need to properly account for:

*

*how many annotations an example received

*how much the annotators disagreed

*the overall quality of each annotator

*the classifier's confidence estimate

*the relative accuracy of the classifier vs individual annotators

