libSVM - probability estimates in multi-class problems

Question

I am using libSVM for classification on a 3-class dataset. Using the option "-b 1" - for getting probability estimates for prediction - gives me surprising results.

I am reproducing the issue with a much simpler dataset here.

The option "-b" can take only two values: "-b 1" denotes we need probability estimates of the predicted label in the output, and "-b 0" indicates we don't need the estimates. "-b 0" is the default i.e. not specifying an option is equivalent to saying "-b 0"

This is my sample training data (let's say the file is called simple):

1 1:0.1 2:0.1 3:0.1
1 1:0.15 2:0.15 3:0.15
2 1:0.5 2:0.5 3:0.5
2 1:0.55 2:0.53 3:0.49
3 1:0.9 2:0.92 3:0.93
3 1:0.88 2:0.91 3:0.97

It's easy to see what I am doing:

• for vectors with each dimension ~0.1, the label is 1.
• for vectors with each dimension ~0.5, the label is 2.
• for vectors with each dimension ~0.9, the label is 3.

Here's my test data (the file is called simple.t):

1 1:0.1 2:0.13 3:0.11
2 1:0.49 2:0.55 3:0.56
3 1:0.9 2:0.95 3:0.99

Commands run with probability enabled:

./svm-train  -b 1 simple
./svm-predict -b 1 simple.t simple.model output
Accuracy = 0% (0/3) (classification)

Output file:

labels 1 2 3
3 0.0447161 0.226854 0.728429
1 0.49332 0.248142 0.258538
1 0.713506 0.24226 0.0442344

Commands run with probability disabled:

./svm-train  simple
./svm-predict simple.t simple.model output
Accuracy = 100% (3/3) (classification)

Output file:

1
2
3

I find this very surprising, if not absurd!

Why is enabling probability estimates changing the way the classifier works? Drastically: the accuracy drops from 100% to 0%. What am I missing here?

Answer 1

I asked Chih-Jen Lin about this problem. Here is his explanation:

This is possible. Internally libsvm's probability outputs conduct a CV procedure. So you need to have enough data. Now your number of data is just too small.

An easy workaround, as stated in their tutorial, is to replicate the data a few times, so that if you had 1 data point for each class, you would now have 5 or 10.

Answer 2

The probability estimation matrix is NOT following the order of 1 2 3 4... the order is probably defined by the sequence of your training labels, whichever comes first. You can check the order inside the 'model' which obtained from the training process. e.g., model.label