38
$\begingroup$

A group of us got to discussing what percentage of a population needs to be tested for COVID-19 in order to estimate the true prevalence of the disease. It got complicated, and we ended the night (over zoom) arguing about signal detection and characteristics of the imagined test. I'm still thinking about it...

So:

  1. Assuming a perfect test, how do you plot the curve of testing reducing the confidence interval around true population infection rate?

  2. Assuming an imperfect test, how do you introduce the signal detection problem of test false positives and negatives?

  3. How do you model all this over time?

I'd love a textbook answer, a reference to a paper (ideally with math, not code), or a convincing argument.

Improve this question
$\endgroup$
5
  • $\begingroup$ For the part where you consider imperfect tests, perhaps see this Q&A. $\endgroup$
    – BruceET
    Commented Apr 16, 2020 at 8:40
  • $\begingroup$ When you say "what percentage of a population" do you mean: we test people who seem ill or have had contact with ill people and publish the results (what most testing currently is). Or we test a representative sample of people and publish that (a small amount of this type testing has been done) $\endgroup$
    – Richard Tingle
    Commented Apr 16, 2020 at 16:13
  • $\begingroup$ Related stats.stackexchange.com/q/456175/35989 $\endgroup$
    – Tim
    Commented Apr 16, 2020 at 18:19
  • 25
    $\begingroup$ As with opinion polls, the sample size isn’t the main issue. The hard bit is getting a sample that is truly representative of the total population. $\endgroup$
    – Mike Scott
    Commented Apr 16, 2020 at 19:41
  • 3
    $\begingroup$ This is one of those cases where defining the population already starts to be an issue. You've got problematic sub-populations like the homeless and illegal immigrants, which are also likely to overlap and be high-risk groups for COVID-19. $\endgroup$
    – MSalters
    Commented Apr 17, 2020 at 12:58

3 Answers 3

Reset to default
33
$\begingroup$

1) Making some assumptions about the population size (namely that it is large enough that a binomial model is appropriate), the prevalence of a disease in a population at a particular time can be obtained by sampling simple random sampling of people and finding who is sick. That is a binomial random variable and the Wald confidence interval for a proportion $p$ is

$$ p \pm 1.96\dfrac{\sqrt{p(1-p)}}{\sqrt{n}}$$

The variance portion is bounded above by 0.5, so we can make the simplifying assumption that the width of the confidence interval is $\sim 2/\sqrt{n}$. So, the answer to this part is that the confidence interval for $p$ decreases like $1/\sqrt{n}$. Quadruple your sample, halve your interval. Now, this was based on using a Wald interval, which is known to be problematic when $p$ is near 0 or 1, but the spirit remains the same for other intervals.

2) You need to look at metrics like specificity and sensitivity.

Sensitivity is the probability that a diseased person will be identified as diseased (i.e. tests positive). Specificity is the probability that a person without the disease is identified as not having the disease (i.e. tests negative). There are lots of other metrics for diagnostic tests found here which should answer your question.

3) I guess this is still up in the air. There are several attempts to model the infection over time. SIR models and their variants can make a simplifying assumption that the population is closed (i.e. S(t) + I(t) + R(t) = 1) and then I(t) can be interpreted as the prevalence. This isn't a very good assumption IMO because clearly the population is not closed (people die from the disease). As for modelling the diagnostic properties of a test, those are also a function of the prevalence. From Bayes rule

$$ p(T+ \vert D+) = \dfrac{P(D+\vert T+)p(T+)}{p(D+)}$$

Here, $P(D+)$ is the prevalence of the disease, so as this changes then the sensitivity should change as well.

Improve this answer
$\endgroup$
9
  • 15
    $\begingroup$ +1. It may be worth emphasizing that the "sampling" to which you refer must be simple random sampling, because it's unlikely any such sample will be obtained, especially in a longitudinal or temporal study ("model this over time"). $\endgroup$
    – whuber ♦
    Commented Apr 16, 2020 at 12:25
  • $\begingroup$ You have two different definitions of specificity, both in bold. Did you mean them that way? $\endgroup$
    – Richard Hardy
    Commented Apr 16, 2020 at 17:45
  • $\begingroup$ @RichardHardyn Whoops. Submit an edit and I will accept. Currently AFK $\endgroup$
    – Demetri Pananos
    Commented Apr 16, 2020 at 17:53
  • $\begingroup$ It's probably worth defining $n$ explicitly here - I was staring at this trying to figure out why the population size mattered (for a sufficiently large population) until I realized that $n$ was probably the sample size. $\endgroup$
    – Milo Brandt
    Commented Apr 17, 2020 at 14:50
  • $\begingroup$ $n$ is defined through the context of the use in a confidence interval. $\endgroup$
    – Demetri Pananos
    Commented Apr 17, 2020 at 15:11
13
$\begingroup$

It has been answered by Dimitri Pananos, I will only add that in order to estimate the prevalence with pre-set precision you need an absolute sample size which is pretty much invariant with the population size (only when the sample is a substantial part of the target population you have a non-negligible finite population correction factor). So there is not a percentage of the population that needs to be tested: 50% of a small population may not be sufficient, 0.5% of a large population may be far enough for the same precision.

Improve this answer
$\endgroup$
10
  • 2
    $\begingroup$ I think a key point made earlier is that sample size in and of itself is not enough. You have to have a sampling frame that allows you to generalize to the population. If you don't have that you could have a very large number of cases, random sample, and your analysis be wrong. I think this tends to get ignored in a lot of discussions of sampling and sample size (and is the really hard thing to get right). $\endgroup$
    – user54285
    Commented Apr 17, 2020 at 0:17
  • 3
    $\begingroup$ @user5285, yes, that's true. But the point is that, even in the case of simple random sampling, asking for the percentage of the population required to sample to obtain a given precision, is wrong. The same logic applies to stratified or more complex sampling designs. Except for very small populations, what matters is not the fraction of the population you are sampling, but the number of units sampled. $\endgroup$
    – F. Tusell
    Commented Apr 17, 2020 at 6:45
  • 2
    $\begingroup$ @F.Tusell What you are saying is that the gain in confidence from sampling 10% of the American population instead of 1% is minimal, while the gain in confidence from increasing from 1% to 10% of a population of 100 (say, a nursing home) is enormous. Or, the other way round, the difference in confidence between testing 10,000 people in NYC vs. the entire U.S. is small (if we simplify and assume a homogeneous population to make the samples representative). $\endgroup$
    – Peter - Reinstate Monica
    Commented Apr 17, 2020 at 9:16
  • 3
    $\begingroup$ Apparently testing 1500 people in each of the 50 largest cities (a total of 75,000 tests) would provide a 98% reliable infection rate that is ± 3% of the actual infection rate. If the proportion turns out to be 10% then we become 99% confident that this is ± 2% of the actual infection rate. $\endgroup$
    – polcott
    Commented Apr 17, 2020 at 16:55
  • $\begingroup$ @polcott, yes that matches my calculations. $\endgroup$
    – F. Tusell
    Commented Apr 17, 2020 at 17:20
1
$\begingroup$

I'll go in a somewhat different direction and say that it depends...

  • Of course any sampling is based on the notion that the sampling is truly random. Trying to account for non-randomness in the sample tremendously complicates the situation.

  • This type of yes/no measurement is non-parametric. Such tests need a larger sample size than if the measurement was parametric.

  • Presumably you're ignoring the problem of false positives and false negatives in the testing. False positives could be a real problem is the disease fraction is low.

  • What is the actual fraction of the diseased? If only 0.1% of the population is diseased then on average one 1 in a 1000 tests would be positive. So the lower the infection rate, the larger the sample would need to be.

  • How precise an estimate do you want? In other words do you want to know the infection rate +/- 20%, or to say +/- 1%. The more precise you want to know the value of the infection rate the larger the sample would need to be.

There is a type of statistical testing called Acceptance Testing which can be used. Basically the important decision is how precise do you want the measurement to be? Then you keep sampling until that level of precision is achieved. So if 50% of the population is infected then a relatively small sample is needed to get to +/- 10% error in the measurement itself (e.g 50% +/- 5%). However if only 0.5% of the population is infected then a much larger sample is needed to determine that the disease level (e.g. 0.5% +/- 0.05%).

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ I disagree with your fourth bullet, the prevalence of the disease has nothing to do with the sample size needed to describe estimate that prevalence. The width of the confidence interval on your estimate depends mainly on sample size. In fact, for equal sample sizes, the closer the prevalence estimate is to 0, the narrower the confidence interval. I suppose this might be true if you're talking about a relative sense of estimating a 0.1% prevalence vs a 0.2% prevalence, but in an absolute sense, prevalence doesn't really affect sample size needed. $\endgroup$
    – Nuclear Hoagie
    Commented Apr 17, 2020 at 13:24
  • 1
    $\begingroup$ @NuclearWang - If the prevalence is 50% how many random samples do you need to have a 95% confidence that you have at least one positive? If the prevalence is 0.5% how many random samples do you need to have a 95% confidence that you have at least one positive? $\endgroup$
    – MaxW
    Commented Apr 17, 2020 at 19:22
  • $\begingroup$ I don't see how "likelihood of finding at least one positive" is related to the precision of the prevalence estimate. With 100 samples, if you find 0 positives, your prevalence confidence interval is [0%, 0.5%]. If you find 50 positives, your prevalence confidence interval is [39.7%, 60.3%]. You don't need to find any positives to have a very good estimate of disease prevalence. In fact, the fewer you find, the better evidence you have that the prevalence is very low. $\endgroup$
    – Nuclear Hoagie
    Commented Apr 17, 2020 at 19:37
  • 1
    $\begingroup$ As of April 14 CDC data showed 661,712 cases in the US. The 2019 population of the US was 328.2 million giving an infection rate of 0.20%. So 100 perfectly chosen random samples would tell you little about the disease prevalence, but it would give you a reasonable estimate of males to females in the US. $\endgroup$
    – MaxW
    Commented Apr 17, 2020 at 20:04
  • 1
    $\begingroup$ @NuclearWang - We're having a violent agreement. My point is that I want to know the value of the measurement itself to with 10% of its value. So for males are 50% +/- 5% of the population. The infection rate is 0.2% +/- 0.02%. My points have been (1) that not knowing what the infection rate is you can't have a predefined a sample size to get a error of +/- 10% of the value. (2) For the data that I gave on sex vs infection rate you'd need a much larger sample for the infection determination. $\endgroup$
    – MaxW
    Commented Apr 17, 2020 at 21:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.