0
$\begingroup$

If out of 500 cases a model finds 20 positives and 480 negatives, but only 10 are true positives, is the model still 98% accurate?

We have this problem of finding "the needles in a hay stack" with text data. True negatives are mostly very easy to find, since not much is "happening" in those texts. Some of the negatives might have false negatives in them, but this is not at the moment the problem and it will get better with new and more precise specification.

The false positives are a bigger problem, because they are showing up as positives and for the person looking at the results, it seems like it's bringing up 50% (=10/20) false results.

So, is the model accuracy 98% or 50%? And is there terminology to support the different metrics? I guess this 50% is the "Predictive value of a positive result", but can it be said that accuracy is 98%? For me it sounds very misleading...

$\endgroup$
1
$\begingroup$

You probably want precision aka $PPV$, which is $TP / (TP + FP)$, e.g. the fraction of true positives in everything your model predicted to be positive. As you mentioned, it's perfectly possible to have low PPV and high accuracy at the same time.

$\endgroup$
  • $\begingroup$ Thanks! So which figure should be reported when describing/marketing the goodness or performance of the model? Is it ok just to talk about accuracy although precision is so low? $\endgroup$ – ElinaJ Feb 9 '15 at 11:35
  • 1
    $\begingroup$ Depends entirely on what is important for you. I can't imagine many settings where you would care about accuracy given such class skew. Probably, precision is much more important for your application. $\endgroup$ – Marc Claesen Feb 9 '15 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.