Let's say I want to train a classifier using supervised learning. I asked a group of human evaluators to decide whether my training samples are positive or negative.
During training, should I i) use samples where a consensus is reached (e.g., a sample receiving 5 positive votes out of 6 votes), or use the votes directly (e.g., if a sample receives 3 positive votes out of 6, we then have effectively 6 samples, 3 positive and 3 negative)?
I consider an election is won if it is decided by a super-majority (e.g., one class winning 75% or more of the votes). A training sample is dropped if no super-majority is reached.
In other words, should I use the outcome of each election or use the individual votes?
In addition, I think I get better performance (in terms of precision and recall using held-back test samples) if I use the election outcomes instead of the individual votes during model training. To summarize,
During training, I use either the election outcome or individual votes to train the classifier. The former outperforms the latter but the there are about half as many training samples using the former method (because many elections did not reach a super-majority).
During testing, I use the election outcome only as ground truth.