I have a classification problem in which I'm using the --ect option for the multi-class algorithm.

The output of the classifier is something as follows:

1.000000 805848386108096
2.000000 133087140195133
2.000000 598100953597523
3.000000 629273927146079
2.000000 547637911979064
1.000000 733923413306849

Where the first part is the class (1 to 3) and the second part my tag/id.

Is there a way to get the 'confidence' level of each prediction? For instance, if the confidence is below a certain threshold, I want to leave the example as "un-classified".

  • $\begingroup$ Does this question help at all: stats.stackexchange.com/questions/88502/… ? Specifically the -oaa flag mentioned in the answer. $\endgroup$ Jun 22 '14 at 1:37
  • $\begingroup$ Thanks for referring that post, but that describes how to do multi-class prediction. It does not resolve the "confidence" / "precision" issue. I would like to get the confidence for each prediction of the multi-class classification. $\endgroup$
    – Roemer
    Jun 22 '14 at 9:07
  • $\begingroup$ Classification is the same thing as prediction on "old" data. $\endgroup$ Jun 22 '14 at 9:28
  • $\begingroup$ I know, but that still does not show an "confidence" level about the classification/prediciton, does it? It only produces integer classes. I would like to know how 'good' the prediction for that example is. Or am I missing something? $\endgroup$
    – Roemer
    Jun 22 '14 at 9:58

Unfortunately, because of the filter tree / elimination implementation in ECT, getting a measure of confidence is not straight-forward. If you can sacrifice some speed, using -oaa with logistic loss and the -r (--raw_predictions) option gives you raw scores that you can convert to a normalized measure of relative "confidence". Say you have a file like this in "ect.dat":

1 ex1| a
2 ex2| a b
3 ex3| c d e
2 ex4| b a
1 ex5| f g

We run the one-against-all:

vw --oaa 3 ect.dat -f oaa.model --loss_function logistic

Then run prediction with raw scores output:

vw -t -i oaa.model ect.dat -p oaa.predict -r oaa.rawp

You get predictions in oaa.predict:

1.000000 ex1
2.000000 ex2
3.000000 ex3
2.000000 ex4
1.000000 ex5

and raw scores for each class in oaa.rawp:

1:0.0345831 2:-0.0888872 3:-0.533179 ex1
1:-0.241225 2:0.170322 3:-0.749773 ex2
1:-0.426383 2:-0.502638 3:0.154067 ex3
1:-0.241225 2:0.170322 3:-0.749773 ex4
1:0.307398 2:-0.387151 3:-0.502747 ex5

You can map these using 1/(1+exp(-score)) and then normalize in various ways to get something like these:

1:0.62144216 2:0.5328338 3:0.20096953 ex1
1:0.57251362 2:0.71125717 3:0.1433303 ex2
1:0.37941591 2:0.29294807 3:0.66095287 ex3
1:0.57251362 2:0.71125717 3:0.1433303 ex4
1:0.72177734 2:0.37525053 3:0.2704246 ex5

Once you have a significantly large data set scored, you can plot threshold in steps of 0.1, for instance, against percent correct if using that threshold to score, to get an idea of what threshold will give you, say, 95% correct for class 1, and so on.

This discussion might be useful.

  • $\begingroup$ I typically do something similar with other classification methods, but I use these as weights against each class's f1 score computed on some large unseen test sample. I can keep updating the test sample and f1 score as new data come in and keep track of drift, rebuilding the models when thresholds or accuracy falls. $\endgroup$
    – wwwslinger
    Jun 27 '14 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.