# How to get confidence on classification predictions with multi-class Vowpal Wabbit

I have a classification problem in which I'm using the --ect option for the multi-class algorithm.

The output of the classifier is something as follows:

1.000000 805848386108096
2.000000 133087140195133
2.000000 598100953597523
3.000000 629273927146079
2.000000 547637911979064
1.000000 733923413306849


Where the first part is the class (1 to 3) and the second part my tag/id.

Is there a way to get the 'confidence' level of each prediction? For instance, if the confidence is below a certain threshold, I want to leave the example as "un-classified".

• Does this question help at all: stats.stackexchange.com/questions/88502/… ? Specifically the -oaa flag mentioned in the answer. Jun 22, 2014 at 1:37
• Thanks for referring that post, but that describes how to do multi-class prediction. It does not resolve the "confidence" / "precision" issue. I would like to get the confidence for each prediction of the multi-class classification. Jun 22, 2014 at 9:07
• Classification is the same thing as prediction on "old" data. Jun 22, 2014 at 9:28
• I know, but that still does not show an "confidence" level about the classification/prediciton, does it? It only produces integer classes. I would like to know how 'good' the prediction for that example is. Or am I missing something? Jun 22, 2014 at 9:58

Unfortunately, because of the filter tree / elimination implementation in ECT, getting a measure of confidence is not straight-forward. If you can sacrifice some speed, using -oaa with logistic loss and the -r (--raw_predictions) option gives you raw scores that you can convert to a normalized measure of relative "confidence". Say you have a file like this in "ect.dat":

1 ex1| a
2 ex2| a b
3 ex3| c d e
2 ex4| b a
1 ex5| f g


We run the one-against-all:

vw --oaa 3 ect.dat -f oaa.model --loss_function logistic


Then run prediction with raw scores output:

vw -t -i oaa.model ect.dat -p oaa.predict -r oaa.rawp


You get predictions in oaa.predict:

1.000000 ex1
2.000000 ex2
3.000000 ex3
2.000000 ex4
1.000000 ex5


and raw scores for each class in oaa.rawp:

1:0.0345831 2:-0.0888872 3:-0.533179 ex1
1:-0.241225 2:0.170322 3:-0.749773 ex2
1:-0.426383 2:-0.502638 3:0.154067 ex3
1:-0.241225 2:0.170322 3:-0.749773 ex4
1:0.307398 2:-0.387151 3:-0.502747 ex5


You can map these using 1/(1+exp(-score)) and then normalize in various ways to get something like these:

1:0.62144216 2:0.5328338 3:0.20096953 ex1
1:0.57251362 2:0.71125717 3:0.1433303 ex2
1:0.37941591 2:0.29294807 3:0.66095287 ex3
1:0.57251362 2:0.71125717 3:0.1433303 ex4
1:0.72177734 2:0.37525053 3:0.2704246 ex5


Once you have a significantly large data set scored, you can plot threshold in steps of 0.1, for instance, against percent correct if using that threshold to score, to get an idea of what threshold will give you, say, 95% correct for class 1, and so on.

This discussion might be useful.

• I typically do something similar with other classification methods, but I use these as weights against each class's f1 score computed on some large unseen test sample. I can keep updating the test sample and f1 score as new data come in and keep track of drift, rebuilding the models when thresholds or accuracy falls. Jun 27, 2014 at 14:29