We are manually classifying texts as positive (+1), neutral (0) or negative (-1). The purpose is to train a sentiment analysis classifier.
We are two people, and we have both classified the same subset (about 500 texts) with a kappa of around 0.8. When we both agree in the same it is clear that it is the "correct" label, but in the case where we do not agree, how to choose the best label? is there any technique to combine two classification labels into a single one?