0
$\begingroup$

Say we have multiple SVMs used in a one-vs-all approach, such that classes a, b, c correspond to 3 SVMs trained positively on the class and then negatively on all other classes.

When we test these we get 3 results. For a class a sample we may get a:0.9, b:0.2, c:0.3.

Because each SVM is computed slightly differently and has different scales, I cannot compare them. Is there an approach that will work on the outputs of one-vs-all classifiers to calibrate the outputs?

I have found a paper which discusses it in section 2.1 (http://www.metarecognition.com/wp-content/uploads/2012/05/cvpr2012_attributes.pdf) but I am unsure if this would work one my one-vs-all approach as the paper seems to be talking about applying this to only one classifier and I have 3.

$\endgroup$

1 Answer 1

2
$\begingroup$

Proposed Solution: Calibration

I have read the "Multi-class" part of your post, but since you are using One vs. All SVMs, i think you should reconsider solving the problem at the binary level. You could calibrate the single svms, so that the resulting output values are comparable.

Calibration methods for the binary SVMs (so that means also applicable in the one vs. all scenario) are Platt scaling1 and Isontonic regression. A nice overview with python code is available here.

For your own use case you would then calibrate each OvA SVM separately and afterwards the calibrated outputs for a, b and c should be comparable.

What does calibration do here?

The key thing here is, that SVMs themselves, are not probabilistic. The output value you mentioned is usually a function of the classified point's distance to the hyperplane. So we are using a heuristic which has no further significance. The goal of this heuristic is that higher numbers are more likely to be the correct result.

You can measure the signifance of your output values using a reliability plot. I will cut short here but essentially you want your reliability curve to be as close as possible to the diagonal. The calibration adds another mapping of output values to calibrated output values. This can handle for example classifiers which have a bias towards high output values. Think of it as another translation step "Ok i got that really confident 0.9 from you classifier A, but i know you always are over-confident so let's make this a 0.5". So a 0.5 value of classifier A should be closer to a 0.5 value of classifier B in the end.

Keep in mind, when using calibration you have to work thoroughly as usual (train/dev/test set).


1. Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3), 61-74.

$\endgroup$
9
  • $\begingroup$ Thank you. I am interested. Can you give more information as to why this works? Why can we calibrate each OVA SVM without regards to the others and then expect them to be comparable? Do we not have to calibrate each OVA SVM with respect to all of the others? $\endgroup$
    – mino
    May 7, 2016 at 22:16
  • 1
    $\begingroup$ My answer got too long, so i edited my post. You only have to calibrate the OVA SVM itself, it still will be OVA (knowing only 2 classes). We expect them to be comparable because using reliability plots we can see, how far we can trust that current output values. If they show good reliability curves (see the post) they will have a better explanatory power. You can also use the curves is advance, to see if your output values are flawed (which was my assumption since you mentioned them not being comparable). $\endgroup$
    – user35825
    May 8, 2016 at 9:58
  • $\begingroup$ That is very helpful. I guess the first thing I should do with the output values is a reliability plot then. They may already be comparable, as you say. Can I just ask (and maybe I'm missing something!) but why does calibrating each OVA SVM separately make them comparable? I would have thought that we need to measure output scores for each OVA SVM and do some adaption in relation to the outputs of the other OVA SVMs? Why do we not need to do this? $\endgroup$
    – mino
    May 8, 2016 at 10:45
  • 1
    $\begingroup$ When you look at the reliability curve's bins, the average output value (confidence) should scale linearly with the average accuracy of the predictions in the same bin. That means on average, the output values should correspond to their probability of being true. When you achieve this property for every OVA SVM they are comparable. $\endgroup$
    – user35825
    May 8, 2016 at 15:58
  • 1
    $\begingroup$ Of course you could do some 'postprocessing' of the combined output, but i think tweaking the binary problem is the easier way. The OVA SVMs only know 2 classes, so the postprocessing would know more classes than the actual classifiers. I had this problem once and i considered calibration my best choice. But i wont rule out that the same effect can be achieved otherwise :). $\endgroup$
    – user35825
    May 8, 2016 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.