Classification score: SVM

Question

I am using libsvm (which is meant for solving binary classification problems) for multi-class classification. How can I get classification scores / confidences for each class to effectively compare them given that libsvm can only produces scores for two classes. Desired output:

Class 1: score1

Class 2: score2

Class 3: score3

Class 4: score4

Answer 1

If I understand your question, you can pass the -b flag with option 1 when building the model like this:

../svm-train -b 1 data_file

And then when creating the prediction vector, you do the same:

../svm-precict -b 1 test_file data_file.model output_file

Here is an output_file example that has 3 classes:

labels 0 -1 1
0 0.635655 0.18753 0.176816

Answer 2

See here: http://en.wikipedia.org/wiki/Support_vector_machine#Multiclass_SVM

I am not familier with libSVM, but this is the way I would do it (Given c classes):

one-vs-all: Train c classifiers, each classifier $i$ with the two classes $c_i$ and (all classes except $c_i$). Afterwards sum-normalize all single-class-scores, i.e. $finalscore(c_i)=\frac{score(c_i)}{\sum{score(c_i)}}$.

one-vs-one: Train (c^2 - c)/2 classifiers $m_{i,j}$ (one for each pair $(c_i,c_j)$. Let $m_{(i,j)}(c_l)$ be the score for class $c_l$ of model $m_{(i,j)}$, which means that $m_{(i,j)}(c_l)>=0$ for $l \in \{i,j\}$, else 0. Afterwards calculate the average score for each class, i.e. $finalscore(c_i)=\frac{\sum_{m_{(l,k)}}m_{(l,k)}(c_i)}{c-1}$ and sum-normalize the finalscores afterwards (as in one-vs-all).

Final remark: This paper (found on libsvm-page) might help, too: A comparison of methods for multi-class support vector machines

Answer 3

If there is the possibility that something may belong to more than one class, another approach is to train N classifiers with the '-b 1' flag (to enable probability estimates). You will then get confidence levels of each data point belonging to each classifier. However, there's still the question of what threshold to use. If you want to get around the problem of picking the 'best' threshold, you can use 11-pt Mean Average Precision. This measures the AP for threshold values [0.0, 0.1, 0.2, ..., 1.0] (thus the 11 pt).

Answer 4

If you are using only linear SVMs, you can try liblinear. It's by the same research group and supports multi-class linear SVM using the '-s 4' option. http://www.csie.ntu.edu.tw/~cjlin/liblinear/

Off-the-shelf Multiclass classifiers for linear SVM and LDA (latent dirichlet allocation) are easily available.

As mentioned on http://en.wikipedia.org/wiki/Multiclass_classification, there are two main approaches for multi class problems for general classifiers: one-vs-all and all-vs-all.

one-vs-all: Train c classifiers for each class: that class versus the rest. Use all c classifiers on a test point, and output the class with the highest score. (Winner Takes All scoring)

all-vs-all: Train a classifier for each pair of classes. Apply each classifier to a test point, and choose the classifier with the highest average score.

Other variations for the final scoring are also possible.