If out of 500 cases a model finds 20 positives and 480 negatives, but only 10 are true positives, is the model still 98% accurate?
We have this problem of finding "the needles in a hay stack" with text data. True negatives are mostly very easy to find, since not much is "happening" in those texts. Some of the negatives might have false negatives in them, but this is not at the moment the problem and it will get better with new and more precise specification.
The false positives are a bigger problem, because they are showing up as positives and for the person looking at the results, it seems like it's bringing up 50% (=10/20) false results.
So, is the model accuracy 98% or 50%? And is there terminology to support the different metrics? I guess this 50% is the "Predictive value of a positive result", but can it be said that accuracy is 98%? For me it sounds very misleading...