1
$\begingroup$

In a system where we perform multi-class classification via a one-vs-all technique, are two scores comparable? E.g.: If I have 0.5 and 0.6 on two different classifiers, is it possible to say that the classifier that has output 0.6 is more likely to relate to the sample than the classifier class that outputs 0.5?

I have trained each classifier on positive training data from one class and all other training data for all other classes as the negative data, as per the standard for one-vs-all.

I'm aware that when comparing two different classifiers in classifying different types of data that the two classifier scores are not comparable because they are calibrated differently with different accuracies, i.e.: a score of 0.6 in one classifier may be a high accuracy for that classifier but a low accuracy for another classifier. I'm wondering whether this applies here and what can be done to get around it?

Improve this question
$\endgroup$
3
  • $\begingroup$ The question discuss framework transformation. You could tag it this way (you used already 5 tags). There is still no framework transformation tag. Why does the question has a SVM tag? $\endgroup$
    – DaL
    May 3, 2016 at 5:48
  • $\begingroup$ Related: Why do one-versus-all multi class SVMs need to be calibrated? (I was reminded of this because someone upvoted my answer earlier today, possibly related to reading this question....) $\endgroup$
    – Danica
    May 3, 2016 at 6:42
  • $\begingroup$ @Dougal It was me that upvoted it! I found it useful but I learnt that they cannot directly be compared without calibration. But I didn't understand how to calibrate them? I can see that Platt's method can be used for scores 1 or 0 but I am using probabilities. Do you know of a method I can use here? Thank you. $\endgroup$
    – mino
    May 3, 2016 at 8:24

2 Answers 2

Reset to default
2
$\begingroup$

It wasn't clear from your original question that you're using a classifier that outputs probabilities. In this case, assuming the probabilities are reasonably well-calibrated, you can directly compare them – that's the main advantage of using a probabilistic framework.

Now, for each item $x$ and class $j$ you essentially have a probability estimate $p_j(x)$ that $x$ is a member of class $j$. Of course, in a standard one-vs-all approach the probabilities most likely will not sum to 1 ($\sum_j p_j(x) \ne 1$), which means they're not actually valid probabilities. It seems reasonable, though, to renormalize them into $\hat p_j = p_j \left( \sum_{j'} p_{j'} \right)^{-1}$, and then treat them as actual probabilities.

If you trust that the probabilities from each classifier are reasonable, then this sum shouldn't be too far from 1, and this is just a "patch" to make them closer to actual probabilities. If you don't trust the probabilities from each classifier, then this of course won't necessarily be any better. I don't know of any theory saying that this has any nice probabilities, but I also don't know of any pitfalls in doing it.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for your answer, it's very informative! Sorry, yes my one-vs-all classifiers output probabilities. How can we make sure probabilities are well-calibrated? I simply create n SVMs with class data and all other data via grid search. In this case can they be compared or will we need additional calibration? $\endgroup$
    – mino
    May 3, 2016 at 22:59
  • $\begingroup$ How are you getting the probabilities from the SVM – Platt's method? Thats probably reasonable; you could also try isotonic regression. You can evaluate the relative calibration by evaluating the cross-entropy on each of the one-vs-all problems. $\endgroup$
    – Danica
    May 4, 2016 at 1:31
  • $\begingroup$ I'll look into this. I've actually using LIBSVM so I don't know how the probabilities are obtained. I'll need to check. I am just worried that because each SVM will be trained on different portions and sections of the same data that it will skew the calibrations and render them incomparable. If you have any more information about this I'd be very grateful if you could provide further information. $\endgroup$
    – mino
    May 4, 2016 at 10:04
  • $\begingroup$ @mino libsvm uses Platt scaling. I think it should be fine, but you could check the cross-entropies $\sum_{i \in \mathrm{test}} y_i \log \hat{p}_i + (1 - y_i) \log( 1 - \hat{p}_i )$ (where $y_i \in \{0, 1\}$) of each prediction. $\endgroup$
    – Danica
    May 4, 2016 at 20:52
  • 1
    $\begingroup$ @zyxue Check out Probability Estimates for Multi-Class Classification by Pairwise Coupling. I believe that this is the standard approach used by LIBSVM and similar. $\endgroup$
    – Danica
    Oct 19, 2017 at 17:16
0
$\begingroup$

In order to compare the classifiers you need to use the same benchmark. I should have chosen the benchmark according to the business need and use a reduction in order to use the classifier used in the different scenario.

If you should predict one of the many values, you should use a dataset in which the concept has this values. The you can use the one-vs-all classifiers you have by returning the result with the highest confidence. Note that this policy of using the classifier is one of many (e.g., you could have choose to return 'Don't know' if you have some results with high confidence). Your usage policy will effect your performance.

In case that you are interested in predicting a single value, you can use your multiple value classifier using a reduction rule that return true if the specific value received the highest confidence.

Now that you have a single benchmark, choose the classifier that best fits your needs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.