6
$\begingroup$

I have a classification problem in which I'm using the --ect option for the multi-class algorithm.

The output of the classifier is something as follows:

1.000000 805848386108096
2.000000 133087140195133
2.000000 598100953597523
3.000000 629273927146079
2.000000 547637911979064
1.000000 733923413306849

Where the first part is the class (1 to 3) and the second part my tag/id.

Is there a way to get the 'confidence' level of each prediction? For instance, if the confidence is below a certain threshold, I want to leave the example as "un-classified".

Improve this question
$\endgroup$
4
  • $\begingroup$ Does this question help at all: stats.stackexchange.com/questions/88502/… ? Specifically the -oaa flag mentioned in the answer. $\endgroup$
    – shadowtalker
    Commented Jun 22, 2014 at 1:37
  • $\begingroup$ Thanks for referring that post, but that describes how to do multi-class prediction. It does not resolve the "confidence" / "precision" issue. I would like to get the confidence for each prediction of the multi-class classification. $\endgroup$
    – Roemer
    Commented Jun 22, 2014 at 9:07
  • $\begingroup$ Classification is the same thing as prediction on "old" data. $\endgroup$
    – shadowtalker
    Commented Jun 22, 2014 at 9:28
  • $\begingroup$ I know, but that still does not show an "confidence" level about the classification/prediciton, does it? It only produces integer classes. I would like to know how 'good' the prediction for that example is. Or am I missing something? $\endgroup$
    – Roemer
    Commented Jun 22, 2014 at 9:58

1 Answer 1

Reset to default
10
+50
$\begingroup$

Unfortunately, because of the filter tree / elimination implementation in ECT, getting a measure of confidence is not straight-forward. If you can sacrifice some speed, using -oaa with logistic loss and the -r (--raw_predictions) option gives you raw scores that you can convert to a normalized measure of relative "confidence". Say you have a file like this in "ect.dat":

1 ex1| a
2 ex2| a b
3 ex3| c d e
2 ex4| b a
1 ex5| f g

We run the one-against-all:

vw --oaa 3 ect.dat -f oaa.model --loss_function logistic

Then run prediction with raw scores output:

vw -t -i oaa.model ect.dat -p oaa.predict -r oaa.rawp

You get predictions in oaa.predict:

1.000000 ex1
2.000000 ex2
3.000000 ex3
2.000000 ex4
1.000000 ex5

and raw scores for each class in oaa.rawp:

1:0.0345831 2:-0.0888872 3:-0.533179 ex1
1:-0.241225 2:0.170322 3:-0.749773 ex2
1:-0.426383 2:-0.502638 3:0.154067 ex3
1:-0.241225 2:0.170322 3:-0.749773 ex4
1:0.307398 2:-0.387151 3:-0.502747 ex5

You can map these using 1/(1+exp(-score)) and then normalize in various ways to get something like these:

1:0.62144216 2:0.5328338 3:0.20096953 ex1
1:0.57251362 2:0.71125717 3:0.1433303 ex2
1:0.37941591 2:0.29294807 3:0.66095287 ex3
1:0.57251362 2:0.71125717 3:0.1433303 ex4
1:0.72177734 2:0.37525053 3:0.2704246 ex5

Once you have a significantly large data set scored, you can plot threshold in steps of 0.1, for instance, against percent correct if using that threshold to score, to get an idea of what threshold will give you, say, 95% correct for class 1, and so on.

This discussion might be useful.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I typically do something similar with other classification methods, but I use these as weights against each class's f1 score computed on some large unseen test sample. I can keep updating the test sample and f1 score as new data come in and keep track of drift, rebuilding the models when thresholds or accuracy falls. $\endgroup$
    – wwwslinger
    Commented Jun 27, 2014 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.