I have read Alexandru Niculescu-Mizil and Rich Caruana's paper "Obtaining Calibrated Probabilities from Boosting" and the discussion in this thread. However, I am still having trouble understanding and implementing logistic or Platt's scaling to calibrate the output of my multi-class boosting classifier (gentle-boost with decision stumps).

I am somewhat familiar with generalized linear models, and I think I understand how the logistic and Platt's calibration methods work in the binary case, but am not sure I know how to extend the method described in the paper to the multi-class case.

The classifier I am using outputs the following:

  • $f_{ij}$ = Number of votes that the classifier casts for class $j$ for the sample $i$ that is being classified
  • $y_i$ = Estimated class

At this point I have the following questions:

Q1: Do I need to use a multinomial logit to estimate probabilities? or can I still do this with logistic regression (e.g. in a 1-vs-all fashion)?

Q2: How should I define the intermediate target variables (e.g. as in Platt's scaling) for the multi-class case?

Q3: I understand this might be a lot to ask, but would anybody be willing to sketch out the pseudo-code for this problem? (on a more practical level, I am interested in a solution in Matlab).

| cite | improve this question | | | | |
  • 1
    $\begingroup$ great question. I have wondered as well about how to construct the calibration even if you do use 1 versus the rest sort of scheme. If you create k models using 1 versus the rest (and there are k classes) do you have to / should you normalize them somehow so that they sum to 1 (e.g. divide each calibrated probability by the sum of all k)? $\endgroup$ – B_Miner Jan 28 '11 at 1:10

This is a topic of practical interest to me as well so I did a little research. Here are two papers by an author that is often listed as a reference in these matters.

  1. Transforming classifier scores into accurate multiclass probability estimates
  2. Reducing multiclass to binary by coupling probability estimates

The gist of the technique advocated here is to reduce the multiclass problem to a binary one (e.g. one versus the rest, AKA one versus all), use a technique like Platt (preferrably using a test set) to claibrate the binary scores/probabilities and then combine these using a techique as discussed in the papers (one is an extenstion of a Hastie et al process of "coupling"). In the first link, the best results were found by simply normalizing the binary probabilities to that they sum to 1.

I would love to hear other advice and if any of these tecnhiqes have been implmented in R.

| cite | improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.