Using Bayes' theorem with given accuracy statistics

Question

A common example to demonstrate the basics Bayes' theorem is that of a drug test or that or a test for a disease. For example, the Wikipedia page for Bayes' Theorem has a example for cannabis testing which discusses the sensitivity and specificity to demonstrate that just because a test can correctly identify an actual user 90% of the time, it doesn't mean that a positive result implies a 90% probability the subject is a user. Basic stuff here, I know.

That all makes sense. But in some cases, I find similar examples where the accuracy is used as a drop-in replacement for sensitivity. That is, the example will be isomorphic to the example above, but the accuracy of the test is used instead of the sensitivity.

I understand that accuracy is the 'true positives' and/plus 'true negatives' divided by all the predictions. It seems like using accuracy as a replacement for sensitivity (or specificity) is an error. For example, if we are talking about death rates, assume that the chance of any living person dying in the next year is roughly 1%. Therefore, I can create a 'death predictor' consisting of: return False and its accuracy will be 99%.

Am I correct to think that applying Bayes' Theorem using accuracy in lieu of sensitivity or specificity is an error? Does using accuracy this way assume that errors are evenly distributed between the two classes?

Hypothetical Example

If I make a mistake here, please let me know.

Let's say we are playing a gambling game with the following rules: A single die will be rolled. Each round you get to choose whether to bet. If you bet and the value on the die is 5 or 6, I give you \$2. If it's any other value, you give me \$1. You may choose to not to bet in which case the round ends and the die is rerolled.

You have a special magic device which predicts whether a 5 or 6 is showing. You've tested this device with the following results:

 TP  | TN  | FP  | FN
----------------------
  4% | 55% | 11% | 30%

Where:

• positive: 5 or 6
• negative: 1 - 4
• TP: true positive
• TN: true negative
• FP: false positive
• FN: false negative

Based on this, we get an

• accuracy: 59%
• sensitivity: 12%
• specificity: 83%

So, on a given roll, if your device says the result is positive, what's the chance that the value is a 5 or 6?

$$ P(A|B) = \frac{P(B|A)P(A)}{P(B)} $$

Using the sensitivity and specificity values we get:

$$ \frac{(.12)(.33)}{(.12)(.33)+(.17)(.66)} = 0.26 $$

That is, if the prediction says it's a 5 or a 6, there's only a 26% chance that it's actually a 5 or a 6.

However, if we plug in 'accuracy' as I've seen it done in some examples, we get:

$$ \frac{(.59)(.33)}{(.59)(.33)+(.41)(.66)} = 0.41 $$

In my completely contrived example, if you use accuracy as your guide for determining whether to play, you will think a positive prediction means a 41% chance you win \$2 and over the course of playing, you should have a positive average return per bet. $$ (.41 * 2) - (.59 * 1) = \$0.23 $$

However, if you (correctly) use the sensitivity and specificity values, you would realize that you should not trust the device's positive predictions because following its positive predictions will give a negative average return on each bet:

$$ (.26 * 2) - (.74 * 1) = \$-0.22 $$

Am I understanding this correctly? I think perhaps some of these examples are being loose with terminology e.g., calling a test "99% accurate" when they really mean "99% sensitive" but I'm also pretty sure a lot of people don't get the distinction.  

Answer 1

Using the accuracy does not return the proper Bayesian expectation.

• Let $A$ be the event of a true 5 or 6
• Let $B$ be the event of the device showing a 5 or 6

You are asking for $P(A|B)$ or the probability that a true 5 or 6 was rolled given the device shows a 5 or 6.

From definitions, we know that $$ P(A|B) = \frac{P(A \cap B)}{P(B)} $$

So we can skip sensitivity and specificity completely and work from the data given.

$P(A \cap B)$ is where the device shows a 5 or 6 AND that actually occurred, which is the true positive rate or $0.04$. $P(B) = P(A \cap B) + P(\tilde{A} \cap B)$ by the law of total probability. The first is the set of events we just described and has probability $0.04$. The second is the set of events in which a five or six is shown on the device but in truth the die rolled a number less than five. This is the False Positive rate or $0.11$. So what we want is $$ \frac{0.04}{0.04+0.11} = \frac{0.04}{0.15} = 0.2\overline{666} $$

Which is what you got using specificity, etc. subject to rounding.

So while you are calculating some statistic using accuracy, it is not the conditional probability of $A$ given $B$.