# How do we decide when a small sample is statistically significant or not?

Sorry if the title isn't clear, I'm not a statistician, and am not sure how to phrase this.

I was looking at the global coronavirus statistics on worldometers, and sorted the table by cases per million population to get an idea of how different countries had fared.

Note My use of Vatican City below is purely because that was the first tiny country I saw in the list. As @smci pointed out, Vatican City has a few issues that may make it different from others. Therefore, please keep "tiny country" in mind when reading on, as my question applies to any tiny country.

The table shows the Vatican City as being the 7th worst country, with 33,666 cases per million. Now given that the total population of Vatican City is only 802, I'm not sure how much we can make of this figure. When the country's population is small, even a minor fluctuation in the number of cases would make a significant difference to the cases per million. As an artificial example, consider a fictional country with only 1 inhabitant. If that person got the virus, then the cases per million would be 1,000,000, which is way higher than anything in that table.

Obviously the Vatican City is an extreme example, but there are other countries with smallish populations that appear quite high on the list, and I guess the same question would apply to them.

So is there a way of deciding what is "too small" a population to be significant?

If this question isn't clear enough, please explain why rather than downvoting, as I would like to understand it, and am happy to clarify if I didn't explain it well enough.

• The case of the Vatican City a) also has a definitional problem, 80+% of people who work there actually live in Italy, it's not like it can close its borders b) what is a "Vatican City case of coronavirus", does it include tourists or visitors? (and exclude people who commute there but went to hospital in Rome?) c) Law of Small Numbers: as you say, "cases per million" is meaningless on a country with a population of only 825, it will either be zero or absurdly high
– smci
Commented Oct 28, 2020 at 0:07
• @smci True, but VC was only an example of a very small country. You could pick another small country where a) and b) were not such major factors. However, point c) is the one that I was trying to explain in my question. Since cases per million will be extreme, it seems we shouldn't take too much notice of the country when comparing. I was trying to get a feel for what size population you would need to make the cases per million meaningful. Commented Oct 28, 2020 at 14:14
• Avrohom ok then you might like to have edited to just say "a tiny country". Otherwise when I see "Vatican City", I see definitional issues a) and b). Similarly, Gibraltar is a territory most of whose workforce are non-resident daily commuters from the south of Spain. And it's going to be really hard to numerically decouple the effect of the magnitude of the commuting population (on infection), from the Law-of-Small-Numbers effect of quoting statistics on the tiny (nominal) population.
– smci
Commented Oct 28, 2020 at 20:04
• @smci Thanks for the observation. I added a note to my question which should (hopefully) clarify Commented Oct 29, 2020 at 14:29

I will describe how a statistician interprets count data. With a tiny bit of practice you can do it, too.

### The basic analysis

When cases arise randomly and independently, the times of their occurrences are reasonably accurately modeled with a Poisson process. This implies that the number of cases appearing in any predetermined interval has a Poisson distribution. The only thing we need to remember about that is that its variance equals its expectation. In less technical jargon, this means that the amount by which the value is likely to differ from the average (its standard error) is proportional to the square root of the average. (See Why is the square root transformation recommended for count data? for an explanation and discussion of the square root and some related transformations of count data.)

In practice, we estimate the average by using the observed value. Thus,

The standard error of a count of independent events with equal expected rates of occurrence is the square root of the count.

(Various modifications of this rule exist for really small counts, especially counts of zero, but that shouldn't be an issue in the present application.)

In the case of Vatican City, a rate of 33,666 cases per million corresponds to

$$\frac{33666}{10^6} \times 802 = 27$$

cases. The square root of $$27$$ is $$5$$ (we usually don't need to worry about additional significant figures for this kind of analysis, which is usually done mentally and approximately).

Equivalently, this standard error is $$\sqrt{27}$$ cases out of $$802$$ people, equivalent to $$6500$$ per million. We are therefore justified in stating

The Vatican City case rate is $$33666\pm 6500$$ per million.

This shows how silly it is to quote five significant figures for the rate. It is better to acknowledge the large standard error by limiting the sig figs, as in

The observed Vatican City case rate is $$34000 \pm 6500$$ per million.

(Do not make the mistake of just taking the square root of the rate! In this example, the square root of 33,666 is only 183, which is far too small. For estimating standard errors square roots apply to counts, not rates.)

A good rule of thumb is to use one additional significant digit when reporting the standard error, as I did here (the case rate was rounded to the nearest thousand and its SE was rounded to the nearest 100).

### A slightly more nuanced analysis

Cases are not independent: people catch them from other people and because human beings do not dart about the world like atoms in a vial of hot gas, cases occur in clusters. This violates the independence assumption. What really happens, then, is that the effective count should be somewhere between the number of cases and the number of distinct clusters. We cannot know the latter: but surely it is smaller (perhaps far smaller) than the number of cases. Thus,

The square root rule gives a lower bound on the standard error when the events are (positively) correlated.

You can sometimes estimate how to adjust the standard error. For instance, if you guess that cases occur in clusters of ten or so, then you should multiply the standard error by the square root of ten. Generally,

The standard error of a count of positively correlated events is, very roughly, the square root of the count times the square root of a typical cluster size.

This approximation arises by assuming all cases in a cluster are perfectly correlated and otherwise the cases in any two different clusters are independent.

If we suspect the Vatican City cases are clustered, then in the most extreme case it is a single cluster: the count is $$1,$$ its square root is $$1,$$ and the standard error therefore is one whole cluster: namely, about $$27$$ people. If you want to be cautious about not exaggerating the reliability of the numbers, then, you might think of this Vatican City rate as being somewhere between just above zero and likely less than 70,000 per million ($$1\pm 1$$ clusters of $$27$$ of out a population of $$802$$).

• Wow, what a great answer! Thanks very much for the clear explanation. So I guess to answer my original question, I'd need to perform the same analysis you did on all the countries, compare the standard error, and then discard any where the SE was large. Would that be correct? Thanks again. Commented Oct 27, 2020 at 14:11
• I don't recommend discarding data, because that's not productive. Use the SE to assess how uncertain each count is. There are (complicated) rules for using the SEs to compare rates. The rules are complicated because the criteria depend on how many comparisons you are making. Very, very roughly, when two rate estimates differ by more than either of their SEs, you will entertain the possibility that the underlying rates are not the same, but otherwise you might want to think of those rates as not being "significantly" different.
– whuber
Commented Oct 27, 2020 at 15:35
• Thanks again, that clarifies it a lot. I'm going to mark your reply as the answer, but would you mind adding your last comment into the answer itself, so it's there in case anyone else reads it. Thanks again. Commented Oct 27, 2020 at 17:03

Quoting Wikipedia:

In statistical hypothesis testing, a result has statistical significance when it is very unlikely to have occurred given the null hypothesis.

Result of a statistical test can be significant, or not. Size of the sample is not a test. Significant in what sense? Prevalence of COVID-19 is a characteristic of particular country, at a particular point of time, the fact that one country has smaller (or larger) prevalence than other country does not make it more, or less "significant". It's like you said that higher people are more significant than short ones, the statement doesn't make sense.

You are correct that smaller sample can vary more than larger one, but you need to consider this relative to the size of the population. The sample of 802 cases would be small for saying something about population of China, but in case of Vatican City, this would be the whole population, so there would be no uncertainty.

Finally, if you mean that prevalence in Vatican City is not "significant" because it does not add up many cases to the total prevalence of COVID-19 around the world, than you are correct. However if this is what you re interested in, than rather then looking at relative prevalence (per 100 000 inhabitants) you should rather look at raw counts, that would obviously be larger, for larger countries.

• I believe the question is trying to get at the concept of standard error, not statistical significance.
– whuber
Commented Oct 26, 2020 at 21:09
• @Tim Thanks for the reply. I guess I didn't explain myself well enough. I am interested in comparing how different countries are faring regarding COVID19, and it seemed to me that the "cases per million" number for a country with a small population would be less reliable than a country with a high population. In my extreme fictitious example, the one person in that wee country could swing the "cases per million" number from zero to 1,000,000 on his/her own. That makes the data for that country less reliable doesn't it? In a slightly bigger country, one family could swing the country's number. Commented Oct 26, 2020 at 21:26
• Very appropriate here: the standard joke about these sketchy normalizations with a small denominator is "the Vatican City has a grand total of 5.26 popes per square mile". Commented Oct 27, 2020 at 15:43
• @FedericoPoloni Like it! However, if you're a statistician, shouldn't you be more precise and say it has 5.2631578947368421052631578947368 popes/sq mi 😀 Commented Oct 27, 2020 at 20:48
• Tim - you say that there would be no uncertainty in this case since we have the entire population yet that doesn't "feel" true. For example assume there is a uniform risk of p = 0.1 of anyone on earth being infected with covid. Then smaller countries would have more variation in the infection rate even though we are looking at entire populations and not just samples. I'd be curious what the correct frequentist approach is here - @whuber computes standard errors but I am not sure what the sampling distribution is here since we are looking at the entire population. Commented Jul 27, 2022 at 19:22

@Avroham. I think the word "significant" is so ambiguous, you shouldn't use it in your question. It has a very definite technial meaning in statistics, but has many other meanings more generally. I think the phrase "statistically convincing" would be better. It is even more ambiguous in one sense, but it doesn't have a technical meaning that can be confused with an every day meaning. @whuber's excellent reply is still totally relevant with this rewording.

• Fair comment, thanks for pointing it out. As I said, I'm not a statistician, so wasn't aware of that. Commented Oct 27, 2020 at 20:46

I think what you're asking is if there is some predetermined minimal sample size that needs to be taken in order to have statistical significance. In the case of looking at the World vs the Vatican in terms of cases/million its obvious with a ratio of 7.8 billion to 807 makes any comparison insignificant. ie, neither is predictive of the other. You want to know what minimal sample size is significant. Is it 780? 7,800? 78,000? 780,000? 7.8 million? 78 M? 780 M? I think you can do small sample sizes when polling voters and get significant results, but with something like covid19 it really does come down to factors such as where, population densities, technological advancement, do they have a modern medical system, etc. On its own, the Vatican sample would probably be a good comparison with a 5 block area of New York City in terms of "could" they see a contraction rate of 33,000+/million? But is it an indication the world will eventually see a contraction rate of 33,000/million? The Vatican sample then is insignificant in a predictive sense. Without the Vatican sample we already know the virus can spread to a whole household and kill everyone in that house. It can also infect everyone in a house and none even show symptoms.

• Yup, you've summed up the question very nicely, and given a good angle on the answer. Thanks Commented Oct 28, 2020 at 14:11