25
$\begingroup$

I have read Alexandru Niculescu-Mizil and Rich Caruana's paper "Obtaining Calibrated Probabilities from Boosting" and the discussion in this thread. However, I am still having trouble understanding and implementing logistic or Platt's scaling to calibrate the output of my multi-class boosting classifier (gentle-boost with decision stumps).

I am somewhat familiar with generalized linear models, and I think I understand how the logistic and Platt's calibration methods work in the binary case, but am not sure I know how to extend the method described in the paper to the multi-class case.

The classifier I am using outputs the following:

  • $f_{ij}$ = Number of votes that the classifier casts for class $j$ for the sample $i$ that is being classified
  • $y_i$ = Estimated class

At this point I have the following questions:

Q1: Do I need to use a multinomial logit to estimate probabilities? or can I still do this with logistic regression (e.g. in a 1-vs-all fashion)?

Q2: How should I define the intermediate target variables (e.g. as in Platt's scaling) for the multi-class case?

Q3: I understand this might be a lot to ask, but would anybody be willing to sketch out the pseudo-code for this problem? (on a more practical level, I am interested in a solution in Matlab).

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ great question. I have wondered as well about how to construct the calibration even if you do use 1 versus the rest sort of scheme. If you create k models using 1 versus the rest (and there are k classes) do you have to / should you normalize them somehow so that they sum to 1 (e.g. divide each calibrated probability by the sum of all k)? $\endgroup$
    – B_Miner
    Commented Jan 28, 2011 at 1:10

1 Answer 1

Reset to default
12
$\begingroup$

This is a topic of practical interest to me as well so I did a little research. Here are two papers by an author that is often listed as a reference in these matters.

  1. Transforming classifier scores into accurate multiclass probability estimates
  2. Reducing multiclass to binary by coupling probability estimates

The gist of the technique advocated here is to reduce the multiclass problem to a binary one (e.g. one versus the rest, AKA one versus all), use a technique like Platt (preferrably using a test set) to claibrate the binary scores/probabilities and then combine these using a techique as discussed in the papers (one is an extenstion of a Hastie et al process of "coupling"). In the first link, the best results were found by simply normalizing the binary probabilities to that they sum to 1.

I would love to hear other advice and if any of these tecnhiqes have been implmented in R.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Links mentioned in answer are outdated. Latest links are: citeseerx.ist.psu.edu/viewdoc/… citeseerx.ist.psu.edu/viewdoc/… $\endgroup$
    – Chandra
    Commented Jul 6, 2018 at 15:42
  • $\begingroup$ Cross reference to here stats.stackexchange.com/questions/362460/… $\endgroup$
    – TMrtSmith
    Commented Aug 16, 2018 at 15:14
  • $\begingroup$ Echoing this answer. This stumped me for some time but the paper by Zadrozny and Elkan proved useful. $\endgroup$
    – songololo
    Commented Jun 27, 2019 at 9:40
  • $\begingroup$ The paper is indeed very useful, but I think this answer doesn't accurately summarize it: in one dataset simple normalization wins, but it depends on which calibration method was used, and for another dataset normalization does not win. The conclusion is still to use simple normalization, because the others don't perform significantly better and it's simple. $\endgroup$
    – Ben Reiniger
    Commented Sep 17, 2021 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.