3
$\begingroup$

I am wondering what the probability of a false positive COVID-19 test would be in my city. I'm attempting to use Bayes Theorem to calculate this, however I'm getting very different results based on how I formulate the problem.

Here are the probabilities given:

  • 1 in 41 people in my city have COVID.
  • The analytical false positive rate of a PCR test is 2 in 1,000
  • The proportion of positive tests is 6.8%.

If I ask the probability of not being infected given a positive test, I get 2.9%.

P(Not Infected given Positive test) = P(Positive test given Not Infected) * P(Not Infected) / P(Positive test) = (.002) * (.976) / (.068) = 2.9%

However if I ask the probability of being infected given a positive test, I get 36%.

P(Infected given Positive test) = P(Positive test given Infected) * P(Infected) / P(Positive test) =(.998) * (.024) / (.068) =36%

Can you set me straight here? I must be making an error. Given a positive test, I would expect the probabilities of being infected or not infected to sum to 1.

A note on the probabilities given:

  • 1 in 41 people infected is based on a model but should be interpreted as the ratio of infected people in the population as a whole, not necessarily the population being tested. I believe it's fair to use this as the pre-test odds of a person being infected if we know nothing else about them.
  • PCR tests are generally described as very specific (i.e., very low false positive rate). Lets assume that of 1000 known covid-free samples, only 2 would return a positive result.
  • 6.8% test positivity rate is based on the number of tests that come back positive. Note that we are not testing the entire population, and you can assume that the population that receives a test is more likely to have covid than the population at large.
$\endgroup$
6
  • $\begingroup$ @Zen can you explain intuitively why that would be the case? And in this specific case, should a person with a positive test be highly sceptical they are not infected or less than confident they are infected? $\endgroup$
    – Erik
    Nov 28, 2020 at 4:32
  • $\begingroup$ $P(A|B) + P(A^c|B) = 1.$ Not necessarily so that $P(A|B) + P(A|B^c).$ Consider $P(A|B) = P(AB)/P(B),$ etc. // Also, perhaps see parts of this Q&A. $\endgroup$
    – BruceET
    Nov 28, 2020 at 5:56
  • $\begingroup$ @Zen: It seems to me the OP is asking why his/her P(Not Infected | Positive test) + P(Infected | Positive test) != 1. I think s/he made a mistake in the calculation (please check if my answer is correct) $\endgroup$
    – dariober
    Nov 28, 2020 at 14:02
  • $\begingroup$ @BruceET: Please see my previous comment to Zen. I think the OP has made a mistake in the calculation $\endgroup$
    – dariober
    Nov 28, 2020 at 14:04
  • $\begingroup$ Your 1st and 3rd note are contradictory. In the 3rd you say "you can assume that the population that receives a test is more likely to have covid than the population at large" but in the 1st you say "...not necessarily the population being tested. I believe it's fair to use this as the pre-test odds of a person being infected if we know nothing else about them" The 6.8% proportion of positive tests show that the prior odds for having covid are not 1/41, and are instead very close to 6.8%. $\endgroup$ Nov 30, 2020 at 1:07

2 Answers 2

1
$\begingroup$

Your first half of the reasoning seems correct.

$$\begin{array}{}P(\text{not infected | positive}) &=& \frac{P(\text{positive | not infected})\cdot P(\text{not infected})}{P(\text{positive})} \\ &=& \frac{0.002 \cdot 0.976}{0.068} \\ &=& 0.0287 \end{array}$$


However we get strange values when we try to deduce $P(\text{positive | infected})$ from

$$\begin{array}{rcl}P(\text{infected | positive}) &=& \frac{P(\text{positive | infected})\cdot P(\text{ infected})}{P(\text{positive})} \\ 0.9713&=& \frac{? \cdot 0.024}{0.068} \end{array}$$

or alternatively from

$$\begin{array}{rcl} P(\text{pos}) &=& P(\text{pos | $\neg$ inf})\cdot P(\text{$\neg$ inf}) + P(\text{pos | inf})\cdot P(\text{ inf})\\ 0.068 &=& 0.002 \frac{40}{41} + P(\text{pos | inf}) \frac{1}{41} \end{array}$$

or

$$P(\text{pos | inf}) = 0.068 \cdot 41 - 0.002\cdot 40 = 2.708$$

Which is not possible. Intuitively we can see this more directly... If only 1 in 41 people are sick (2.43 %)then it is strange to find that 6.8% of the tests are positive. The percentage of positive tests is much higher than the people that are sick.

This can be explained in two ways:

  • The false positive rate is not correct (it should be much higher and somewhere close to this 6.8%).

    However, if this is a realistic example about Covid-19 testing then the false positive rate is probably not so high (unless something went very wrong).

  • Among the people that are performing tests there are a higher proportion of sick people (higher than 1 in 41).

    If you would know that $P(\text{pos | $\neg$ inf}) = 0.002$ and $P(\text{pos | inf}) = 0.998$ then you could deduce the prior $P(\text{$\neg$ inf})$ and $P(\text{inf})$ from

    $$\begin{array}{rccccccc} \underbrace{P(\text{pos})}_{0.068} &=& \underbrace{P(\text{pos | $\neg$ inf})}_{0.002}\cdot P(\text{$\neg$ inf}) + \underbrace{P(\text{pos | inf})}_{0.998}\cdot \underbrace{P(\text{ inf})}_{1-P(\text{$\neg$ inf})}\\ \end{array}$$

    which leads to

    $$P(\text{$\neg$ inf}) = \frac{0.998-0.068}{0.996} \approx 0.934$$

    and

    $$P(\text{$\neg$ inf| pos}) = \frac{0.002 \cdot \frac{0.998-0.068}{0.996}}{0.068} \approx 0.0275 $$

$\endgroup$
5
  • $\begingroup$ The OP may need to clarify, but my guess is that 0.002 is P(negative test | infected) and 6.8% is the % positive test across all the tested people. From that, you can get P(positive test | not infected) and things seem to make sense to me (see my answer) $\endgroup$
    – dariober
    Nov 28, 2020 at 21:01
  • $\begingroup$ @dariober that is one way how you can fix the numbers. But I believe that the (0.068×41 − 0.998)÷40 = 4.475% false positive error rate is a bit high for covid-19 tests. Another way is that the prior is not correct and more than 1:41 of the people are sick. (The OP need to clarify where these numbers come from, but I can imagine that the 1:41 of infected people in the city is not the same has the fraction of infected people that perform a test) $\endgroup$ Nov 28, 2020 at 21:12
  • $\begingroup$ I've clarified the probabilities given in the original post to address your comments here. $\endgroup$
    – Erik
    Nov 28, 2020 at 23:17
  • $\begingroup$ I think you've identified the issue that among the people tested we have a higher proportion of sick people. Thus the probabilities given are for different populations and the math doesn't square. Interestingly, in further research I found a site for clinicians trying to answer the same question. It uses a likelihood ratio to calculate post-test odds. It returns a 92% chance of having covid given a positive test. calculator.testingwisely.com/playground/2.4/95/99.8/positive $\endgroup$
    – Erik
    Nov 30, 2020 at 15:27
  • $\begingroup$ The FDA has provided information about the issue of false positive rapid antigen tests to detect SARS-CoV-19 infection and the technical problems in testing that may contribute to false positive antigen tests. The FDA discusses the problem of false positive tests in settings with a low prevalence of infection (low prior probability of being infected at that moment). fda.gov/medical-devices/letters-health-care-providers/… $\endgroup$ Nov 30, 2020 at 18:07
1
$\begingroup$

NOTE this answer precedes the OP's note about percentages. This answer assumed:

  • 1/41 is the prior probability of infection in the tested population (not the population as a whole)

  • 2/1000 is the false negative rate of the PCR test $P(PCR^{-}|Infected)$


I may be confused myself as my calculations do give probabilities summing to 1.

The analytical false positive rate of a PCR test is 2 in 1,000

I assumed you mean false negative rate, right? Otherwise how can you have 2.4% true positives (1/41), 6.8% positive tests but only 0.2% false positives?

Here's my reasoning. First let's set up a simple tree with 100k people (there is some rounding here):

                      100000 people
                           |
                 --------------------
                 |                  | 
              Infected          Not infected 
              P = 1/41           P = 40/41  
               2439                97561
                 |                   |
         -------------          ------------
         |           |          |          |
       PCR+        PCR-        PCR+       PCR-
     P = 0.998   P = 0.002   P = 0.044   P = 0.956
       2434          5         4366       93195

Here P = 0.044 is the false positive rate and comes from 0.068 - (0.998 * 1/41).

And here's the probabilities:

$$ P(inf^{-}|PCR^{+}) = \frac{P(inf^{-}) \cdot P(PCR^{+}|inf^{-})}{P(inf^{-}) \cdot P(PCR^{+}|inf^{-}) + P(inf^{+}) \cdot P(PCR^{+}|inf^{+})} \\ = \frac{(40/41) \cdot 0.044}{(40/41) \cdot 0.044 + (1/41) \cdot 0.998} = 0.638 $$

and

$$ P(inf^{+}|PCR^{+}) = \frac{P(inf^{+}) \cdot P(PCR^{+}|inf^{+})}{P(inf^{-}) \cdot P(PCR^{+}|inf^{-}) + P(inf^{+}) \cdot P(PCR^{+}|inf^{+})} \\ = \frac{(1/41) \cdot 0.998}{(40/41) \cdot 0.044 + (1/41) \cdot 0.998} = 0.362 $$

If I'm correct, your mistake is in using 0.068 as denominator in the Bayes formulas.

$\endgroup$
3
  • $\begingroup$ I appreciate your approach here, however I'm confused by the results. These results would suggest that the PCR test is actually wrong more than it is right (i.e., Pr(Infected given a positive test) is only 36.2%). $\endgroup$
    – Erik
    Nov 28, 2020 at 16:42
  • 1
    $\begingroup$ I should also explain the probabilities given a bit. In my city, we are not testing every single person. However, we assume (based on a model) that 1 in 41 people is infected. Thus, if we grabbed 41 people at random, we would expect one to be infected. In fact, people present for testing based on their (or their doctor's) concern that they are in fact infected, or because they want to justify traveling, or some other reason. Thus, the percent of positive tests does not reflect the background prevalence of COVID. Obviously this is introducing some bias that we are not fully accounting for. $\endgroup$
    – Erik
    Nov 28, 2020 at 16:48
  • 1
    $\begingroup$ @Erik Pr(Infected | positive test) is lower than Pr(Not Infected | positive test) because there are many more non infected than infected people. So even if the false positive rate is only 4.4%, the number of false positives outnumbers the true positives. Regarding your second comment, I assumed that 1/41 is the expected frequency of infected people that do get tested (not necessarily the frequency in the whole population) and 1/41 is before seeing the test results. Does it make sense? $\endgroup$
    – dariober
    Nov 28, 2020 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.