How many tests should we do to estimate the percentage of people who contracted COVID-19 in Lombardy? Lombardy, a region of Italy, registered many severe cases of COVID-19 in the last few months.
Unfortunately the available data don't allow us to estimate the percentage P of people who contracted the virus in Lombardy (please don't dispute this, but take it as an assumption).
Now, suppose we have a medical test that says wether a person contracted the virus or not (EDIT: let's assume that the test always gives the correct answer). How many people should we test to estimate P with sufficiently small error?
Data are:


*

*10 million people live in Lombardy.

*P can be assumed to be at least 4 %.

*No assumption can be made about the uniform distribution of P, neither geographically, nor by age, gender or whatever.

*A satisfactory error percent on P would be 25%.


A related question is: how could we confirm the estimated error on P once we have the data? For example, could we bootstrap the data?
Thanks
 A: This is actually a handbook example of determining the sample size needed for estimating binomial proportion (e.g. Jones et al, 2004, Naing, 2003 for other references and examples).
First of all, to make it more precise, we are talking about finding such sample size, that with probability $\alpha$, the difference between the true probability of being infected $p$ and it's estimate $\hat p$ is not greater then $(100\times\delta\,)\%$
$$
\Pr(|p - \hat p| \le \delta p) = \alpha
$$
Given that the target population is large, we would usually assume binomial distribution to represent it, i.e. we say that it is large enough, that the chance of randomly sampling someone more then once is negligible. The distribution is parametrized by probability of "success" (here, probability of being infected) $p$ and the number of samples we draw $n$. Let's denote the observed number of infected people as $k$, in such case, $\hat p = k/n$ is the fraction of infected people in the sample and we treat it as an estimate of the number of infected people in the whole population. If we wanted to calculate confidence interval for $\hat p$, we could use normal approximation
$$
\hat p \pm z_\alpha \sqrt{\frac{\hat p(1-\hat p)}{n}}
$$
where $z_\alpha$ is the ordinate from standard normal distribution, where for $z$ drawn from standard normal distribution we have $\Pr(-z_\alpha < z < z_\alpha) = \alpha$. You are saying, that you'd like to see this interval to be equal to $\hat p \pm \delta p$. As discussed in the linked resources, you can solve this, so that for given $p$, precision $\delta$, and certanity $\alpha$, you can guesstimate the sample size needed
$$
n \approx \Big(\frac{z_\alpha}{\delta p}\Big)^2 \; p(1-p)
$$
Assuming $(100 \times \alpha)\% = 99\%$ confidence interval, we can plot this for different values of $p$, to find out that for $100 \times p > 4 \%$ the needed sample sizes are generally not much larger then $2000$ samples.

For example, for $p=0.04$ ($4\%$ infected) this yields:

> z <- function(alpha) qnorm(alpha)
> n <- function(p, alpha=0.99, delta=0.25) (z(alpha)/(p*delta))^2 * p*(1-p)
> n(0.04)
[1] 2078.167

To convince yourself, you can verify this by simulation, where you would draw $n$ samples from binomial distribution with probability of infection $p$, repeat this procedure $R$ times, and then verify how often was your result not further then $(100 \times \delta) \%$ from the true value:
> set.seed(123)
> sim <- function(p, n, delta, nsim=50000) mean(abs(p - rbinom(nsim, n, p)/n) / p <= delta)
> sim(0.04, 2078, 0.25)
[1] 0.97858

So we wanted to be $99\%$ sure and the approximation gives us, while the in the simulation, in $97.8\%$ cases the result was within the interval. Not bad.
Notice that this is just a simple approximation for the calculation, assuming simple random sampling. In case of whole population locked in their houses, sampling individuals at random may be not as hard as in case of most of the usual surveys. On another hand, things may not go as smooth as planned, or you may be willing to use other sampling schema, to have higher chance for it being representative, what would make calculating it more complicated. Moreover, the tests used aren't perfect and give false results as described, for example by New York Times, or Washington Post, and you'd need to account for that as well. Also you need to remember, there were many examples where such simple problems get more complicated then expected, e.g. social surveys on Trump's support before the election got very wrong, nonetheless that they used state of art survey methodology.
