4
$\begingroup$

I am using libsvm (which is meant for solving binary classification problems) for multi-class classification. How can I get classification scores / confidences for each class to effectively compare them given that libsvm can only produces scores for two classes. Desired output:

Class 1: score1

Class 2: score2

Class 3: score3

Class 4: score4
|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ Can you please elaborate? This question makes no sense in the current form. $\endgroup$ – user88 Jun 18 '11 at 17:11
  • $\begingroup$ I do not want a single class id as the classification output. Rather the output should be like: the given sample can be classified to class1 with probability score1. $\endgroup$ – Xolve Jun 18 '11 at 18:28
  • 1
    $\begingroup$ I edited your question according to your comment. Please check (after the edit has approved) that it still reflects your problem and intention. $\endgroup$ – steffen Jun 20 '11 at 6:23
3
$\begingroup$

If I understand your question, you can pass the -b flag with option 1 when building the model like this:

../svm-train -b 1 data_file

And then when creating the prediction vector, you do the same:

../svm-precict -b 1 test_file data_file.model output_file

Here is an output_file example that has 3 classes:

labels 0 -1 1
0 0.635655 0.18753 0.176816
|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Nice! What happens internally when this flag is set (i.e. how are the confidences estimated) ? $\endgroup$ – steffen Jun 20 '11 at 15:37
  • $\begingroup$ It's probably buried in the function, but I haven't investigated it. $\endgroup$ – Milktrader Jun 20 '11 at 20:03
  • $\begingroup$ Hmm, 0.635655 + 0.18753 + 0.176816 = 1.000001, could just be ~ class sizes ? If you want a Confusion matrix or its diagonal, split train / test and look at Confusion_matrix( test ). (If you k-fold cross-validate, average the k confusion matrices ?) $\endgroup$ – denis Oct 21 '11 at 10:35
3
$\begingroup$

See here: http://en.wikipedia.org/wiki/Support_vector_machine#Multiclass_SVM

I am not familier with libSVM, but this is the way I would do it (Given c classes):

one-vs-all: Train c classifiers, each classifier $i$ with the two classes $c_i$ and (all classes except $c_i$). Afterwards sum-normalize all single-class-scores, i.e. $finalscore(c_i)=\frac{score(c_i)}{\sum{score(c_i)}}$.

one-vs-one: Train (c^2 - c)/2 classifiers $m_{i,j}$ (one for each pair $(c_i,c_j)$. Let $m_{(i,j)}(c_l)$ be the score for class $c_l$ of model $m_{(i,j)}$, which means that $m_{(i,j)}(c_l)>=0$ for $l \in \{i,j\}$, else 0. Afterwards calculate the average score for each class, i.e. $finalscore(c_i)=\frac{\sum_{m_{(l,k)}}m_{(l,k)}(c_i)}{c-1}$ and sum-normalize the finalscores afterwards (as in one-vs-all).

Final remark: This paper (found on libsvm-page) might help, too: A comparison of methods for multi-class support vector machines

|cite|improve this answer
$\endgroup$
2
$\begingroup$

If there is the possibility that something may belong to more than one class, another approach is to train N classifiers with the '-b 1' flag (to enable probability estimates). You will then get confidence levels of each data point belonging to each classifier. However, there's still the question of what threshold to use. If you want to get around the problem of picking the 'best' threshold, you can use 11-pt Mean Average Precision. This measures the AP for threshold values [0.0, 0.1, 0.2, ..., 1.0] (thus the 11 pt).

|cite|improve this answer
$\endgroup$
1
$\begingroup$

If you are using only linear SVMs, you can try liblinear. It's by the same research group and supports multi-class linear SVM using the '-s 4' option. http://www.csie.ntu.edu.tw/~cjlin/liblinear/

Off-the-shelf Multiclass classifiers for linear SVM and LDA (latent dirichlet allocation) are easily available.

As mentioned on http://en.wikipedia.org/wiki/Multiclass_classification, there are two main approaches for multi class problems for general classifiers: one-vs-all and all-vs-all.

one-vs-all: Train c classifiers for each class: that class versus the rest. Use all c classifiers on a test point, and output the class with the highest score. (Winner Takes All scoring)

all-vs-all: Train a classifier for each pair of classes. Apply each classifier to a test point, and choose the classifier with the highest average score.

Other variations for the final scoring are also possible.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.