4
$\begingroup$

A common example to demonstrate the basics Bayes' theorem is that of a drug test or that or a test for a disease. For example, the Wikipedia page for Bayes' Theorem has a example for cannabis testing which discusses the sensitivity and specificity to demonstrate that just because a test can correctly identify an actual user 90% of the time, it doesn't mean that a positive result implies a 90% probability the subject is a user. Basic stuff here, I know.

That all makes sense. But in some cases, I find similar examples where the accuracy is used as a drop-in replacement for sensitivity. That is, the example will be isomorphic to the example above, but the accuracy of the test is used instead of the sensitivity.

I understand that accuracy is the 'true positives' and/plus 'true negatives' divided by all the predictions. It seems like using accuracy as a replacement for sensitivity (or specificity) is an error. For example, if we are talking about death rates, assume that the chance of any living person dying in the next year is roughly 1%. Therefore, I can create a 'death predictor' consisting of: return False and its accuracy will be 99%.

Am I correct to think that applying Bayes' Theorem using accuracy in lieu of sensitivity or specificity is an error? Does using accuracy this way assume that errors are evenly distributed between the two classes?

Hypothetical Example

If I make a mistake here, please let me know.

Let's say we are playing a gambling game with the following rules: A single die will be rolled. Each round you get to choose whether to bet. If you bet and the value on the die is 5 or 6, I give you \$2. If it's any other value, you give me \$1. You may choose to not to bet in which case the round ends and the die is rerolled.

You have a special magic device which predicts whether a 5 or 6 is showing. You've tested this device with the following results:

 TP  | TN  | FP  | FN
----------------------
  4% | 55% | 11% | 30%

Where:

  • positive: 5 or 6
  • negative: 1 - 4
  • TP: true positive
  • TN: true negative
  • FP: false positive
  • FN: false negative

Based on this, we get an

  • accuracy: 59%
  • sensitivity: 12%
  • specificity: 83%

So, on a given roll, if your device says the result is positive, what's the chance that the value is a 5 or 6?

$$ P(A|B) = \frac{P(B|A)P(A)}{P(B)} $$

Using the sensitivity and specificity values we get:

$$ \frac{(.12)(.33)}{(.12)(.33)+(.17)(.66)} = 0.26 $$

That is, if the prediction says it's a 5 or a 6, there's only a 26% chance that it's actually a 5 or a 6.

However, if we plug in 'accuracy' as I've seen it done in some examples, we get:

$$ \frac{(.59)(.33)}{(.59)(.33)+(.41)(.66)} = 0.41 $$

In my completely contrived example, if you use accuracy as your guide for determining whether to play, you will think a positive prediction means a 41% chance you win \$2 and over the course of playing, you should have a positive average return per bet. $$ (.41 * 2) - (.59 * 1) = \$0.23 $$

However, if you (correctly) use the sensitivity and specificity values, you would realize that you should not trust the device's positive predictions because following its positive predictions will give a negative average return on each bet:

$$ (.26 * 2) - (.74 * 1) = \$-0.22 $$

Am I understanding this correctly? I think perhaps some of these examples are being loose with terminology e.g., calling a test "99% accurate" when they really mean "99% sensitive" but I'm also pretty sure a lot of people don't get the distinction.  

$\endgroup$
11
  • $\begingroup$ The context defines the usefulness of any statistics. E.g., if I give you \$1 for each correct prediction, the prediction accuracy suddenly becomes useful. In addition, it often helps to illustrate a definition with an example. Maybe this is what motivated the examples you are referring to. Thus, I feel that the statement "using accuracy in lieu of sensitivity or specificity is an error" is too general to be correct ;) $\endgroup$
    – Semoi
    Commented Dec 29, 2023 at 8:24
  • $\begingroup$ @Semoi I'm not sure I see your point. Accuracy includes both correct positives and correct negatives. It's possible to have a 'high accuracy' approach that produces mostly correct negative predictions but is essentially random (or worse) when it comes to correct positive predictions. In such a situation, using that high accuracy value when the subject is correct predictions would seem to be invalid. And no, I don't think it would be valuable in your example. If we were 1-1 betting, I would be nearly guaranteed to lose my shirt. $\endgroup$
    – JimmyJames
    Commented Dec 29, 2023 at 16:50
  • $\begingroup$ @Semoi I added an example for clarifying what I am asking about. $\endgroup$
    – JimmyJames
    Commented Dec 29, 2023 at 19:39
  • $\begingroup$ If you get a buck when you’re right and lose a buck when you’re wrong, you should expect to make money with an accuracy higher than $50\%$. There might be a better strategy if there is imbalance, but you don’t lose your shirt just because there is a better strategy. $\endgroup$
    – Dave
    Commented Dec 29, 2023 at 19:45
  • $\begingroup$ @Dave But if you are always betting on positive results, and your sensitivity is less than 50%, you would lose money, no? $\endgroup$
    – JimmyJames
    Commented Dec 29, 2023 at 19:50

1 Answer 1

4
$\begingroup$

Using the accuracy does not return the proper Bayesian expectation.

  • Let $A$ be the event of a true 5 or 6
  • Let $B$ be the event of the device showing a 5 or 6

You are asking for $P(A|B)$ or the probability that a true 5 or 6 was rolled given the device shows a 5 or 6.

From definitions, we know that $$ P(A|B) = \frac{P(A \cap B)}{P(B)} $$

So we can skip sensitivity and specificity completely and work from the data given.

$P(A \cap B)$ is where the device shows a 5 or 6 AND that actually occurred, which is the true positive rate or $0.04$. $P(B) = P(A \cap B) + P(\tilde{A} \cap B)$ by the law of total probability. The first is the set of events we just described and has probability $0.04$. The second is the set of events in which a five or six is shown on the device but in truth the die rolled a number less than five. This is the False Positive rate or $0.11$. So what we want is $$ \frac{0.04}{0.04+0.11} = \frac{0.04}{0.15} = 0.2\overline{666} $$

Which is what you got using specificity, etc. subject to rounding.

So while you are calculating some statistic using accuracy, it is not the conditional probability of $A$ given $B$.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks. I take this to mean my assertion is correct in general that using accuracy is not appropriate and that you really need more detail to determine the conditional probability. $\endgroup$
    – JimmyJames
    Commented Dec 29, 2023 at 20:35
  • 1
    $\begingroup$ Yes. Consider that accuracy also incorporates the case of the true negatives, which has no bearing on the question of Actual Disease given Indicated Disease, or True Positive, since the universe in which we are interested only contains events where the Indication is True. $\endgroup$
    – Avraham
    Commented Dec 31, 2023 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.