I'm classifying text using the one-vs-all approach. There are three classes. I've trained 3 different binary SVM classifiers using 10-fold cross-validation. The accuracy of the binary classifiers is fairly high (above 0.8 for all three classes). Where I'm stuck is utilizing the result of the 3 classifiers to output the most likely class for a sample. Since it's not meaningful to compare the classifier outputs directly, I'm using Platt's technique (PDF is available) to calibrate them. The graph below shows the calibration results:
The top row is the histogram of the classifier outputs. The bottom row is the posterior probability for each example. This graph is analogous to Figures 1 and 2 of Platt's paper.
Since the sigmoid fits look good, I think I've implemented the fitting algorithm *described by the pseudo-code at the end of the paper) correctly. However, I don't get good classification results in the end. Specifically, there are too many "class 3" false positive. There are almost none "class 2" true positives.
My questions are:
- Can anybody see what could be wrong?
- What should I try next? Should I scrap one-vs-all and do one-vs-one?
- Does one-vs-one require a similar type of calibration? Or can you just use the sign of the SVM output to determine the winner of a particular vote?