In a system where we perform multi-class classification via a one-vs-all technique, are two scores comparable? E.g.: If I have 0.5 and 0.6 on two different classifiers, is it possible to say that the classifier that has output 0.6 is more likely to relate to the sample than the classifier class that outputs 0.5?
I have trained each classifier on positive training data from one class and all other training data for all other classes as the negative data, as per the standard for one-vs-all.
I'm aware that when comparing two different classifiers in classifying different types of data that the two classifier scores are not comparable because they are calibrated differently with different accuracies, i.e.: a score of 0.6 in one classifier may be a high accuracy for that classifier but a low accuracy for another classifier. I'm wondering whether this applies here and what can be done to get around it?