A smaller dataset is better: Is this statement false in statistics? How to refute it properly?

Question

Dr. Raoult, who promotes Hydroxychloroquine, has some really intriguing statement about statistics in the biomedical field:

It's counterintuitive, but the smaller the sample size of a clinical test, the more significant its results are. The differences in a sample of 20 people may be more significant than in a sample of 10,000 people. If we need such a sample, there is a risk of being wrong. With 10,000 people, when the differences are small, sometimes they don't exist.

Is this a false statement in statistics? If so, is it therefore also false in the Biomedical field? On which basis can we refute it properly, by a confidence interval?

Dr. Raoult promotes Hydroxychloroquine as a cure for Covid-19, thanks to an article about data from 24 patients. His claims have been repeated a lot, but mainly in the mainstream media, not in the scientific press.

In machine learning, the SciKit workflow states that before choosing any model, you NEED a dataset with at least 50 samples, whether it be for a simple regression, or the most advance clustering technique, etc., which is why I find this statement really intriguing.

---

EDIT: some of the answers below make the assumption of no result bias. They deal with the concept of power and effect size. However it seems there is a bias in Dr. Raoult data. The most striking being removing data for the dead, for the reason they could not provide data for the entire duration of the study.

My question remains however focused on the impact of using a small sample size.

• Source of the statement about Statistics in a French magazine
• Reference to the scientific paper in question.

Answer 1

I agree with many of the other answers here but think the statement is even worse than they make it out to be.

The statement is an explicit version of an implicit claim in many shoddy analyses of small datasets. These hint that because they have found a significant result in a small sample, their claimed result must be real and important because it is 'harder' to find a significant effect in a small sample. This belief is simply wrong, because random error in small samples means that any result is less trustworthy, whether the effect size is large or small. Large and significant effects are therefore more likely to be of the incorrect magnitude and more importantly, they can be in the wrong direction. Andrew Gelman refers to these usefully as 'Type S' errors (estimates whose sign is wrong) as opposed to 'Type M' errors (estimates whose magnitude is wrong). Combine this with the file-drawer effect (small, non-significant results go unpublished, while large, significant ones are published) and you are most of the way to the replication crisis and a lot of wasted time, effort and money.

Thanks to @Adrian below for digging up a figure from Gelman that illustrates this point well:

This may seem to be an extreme example but the point is entirely relevant to the argument made by Raoult.

Answer 2

"It's counterintuitive, but the smaller the sample size of a clinical test, the more significant its results are. The differences in a sample of 20 people may be more significant than in a sample of 10,000 people. If we need such a sample, there is a risk of being wrong. With 10,000 people, when the differences are small, sometimes they don't exist."

I have read the linked article (via Google-translate) in which this quote is given. Unfortunately it does not give any additional clarity of what Prof Raoult meant. Taken on its own, this statement makes no sense at all to me. It is a jumble of unclear references and invalid reasoning, and it exhibits a fundamental misunderstanding of the goal of statistical inference and the mechanics of a hypothesis test. The goal of sampling is not to try to trick the significance test; it is to make the most accurate inference possible about an unknown parameter or hypothesis, and that is done by taking as much data as possible.

As to the claim that a lower sample size will tend to be "more significant", that is false. Assuming you are dealing with continuous data, and your test assumptions are correct, the p-value of the hypothesis test should be uniform under the null hypothesis regardless of the sample size --- i.e., the formula for the p-value takes account of the sample size and so there is no tendency for smaller samples to be "more significant". If there were such a tendency, this would be considered a failure of the testing procedure, not something to try to take advantage of in order to "trick" the hypotheses test.

Prof Raoult states that we may "need such a sample" (i.e., a sample with a significant difference), which unfortunately suggests that the goal of the test methodology is to maximise the chances of coming to a pre-conceived desirable conclusion. This is the kind of thing I hear occasionally from applied researchers who get too involved in trying to prove some hypothesis of theirs, and it makes me cringe --- if the goal of statistical testing is merely to affirm a pre-conceived conclusion then we might as well jettison statistics altogether.

Now, it is possible that Prof Raoult had an entirely different point in mind, and he is simply mashing up his statistical words and saying the wrong thing. (The last sentence is contradictory as written, so he must obviously mean somthing else, but I don't know what.) I have seen that happen many times before when hearing descriptions of statistical phenomena from applied researchers who have no training in theoretical statistics. In this case, I would just ignore the quote, because it is either flat-out wrong, or it is a failed attempt to say something completely different. In either case, you are right in your suspicions --- it is not better to have less data.

Answer 3

(I think the phrasing is deliberately provocative.)

If you have 10 observations and want to show that their mean is not zero, it will have to be quite a bit different from 0 if you want to have any kind of chance (power) of detecting the difference.

If you have a trillion observations and want to show that their mean is not 0, the mean could be just a tiny bit different from zero, perhaps just 0.01, and you would still have a considerable chance of detecting this difference. Yes, we all agree that $0\ne 0.01$, but the practical significance of a mean of 0.01 instead of 0 might be inconsequential: no one cares.

If you detect a difference in that sample of ten, however, the difference from zero is likely to be quite great, probably into the realm of having practical significance. The quote is about practical significance.

Power calculations along with subject matter expertise guiding what counts as an interesting difference (“effect size”) formalize this.

EDIT

The promised power calculation, which turned out to be an effect size calculation.

library(pwr)
n1 <- 100
n2 <- 100000
alpha = 0.05
power = 0.8

# find the effect size, d, for n=100
#
pwr.t.test(n1, sig.level=alpha, power=power)$d # this is 0.3981407

# find the effect size, d, for n=100,000
#
pwr.t.test(n2, sig.level=alpha, power=power)$d # this is 0.01252399

In this example, the test on only 100 subjects is able to detect a difference in mean of $0.398$ $80\%$ of the time. The test on 100,000 subjects is able to detect a difference of $0.013$ $80\%$ of the time. If you need a difference of at least $0.15$ in order for the findings to be interesting, then it isn't so helpful to get the 100,000-subject test going "ding ding ding, REJECT" every time it sees an observed effect of $0.013$. However, if the 100-subject test rejects, you can have more confidence that the effect size is large enough to be interesting.

(That difference is number of standard deviations of the population of the group.

Answer 4

Can you confirm that it is a FALSE statement in statistics

I think the statement is phrased poorly. In this context, the word "significant" seems to have the flavor of "importance". Difference detected in smaller datasets are not somehow more important or meaningful by virtue of being detected in small datasets. Rather, differences detected in small datasets are often times very large when compared to the inherent noise in the data (assuming the differences are not false positives or the result of some sort of bias), explaining why we detected them in the first place. The term "significant" has been overloaded in statistics, which often leads to confusion and misuse.

I would not conclude from this that smaller datasets are better. Indeed, large data (or perhaps more appropriately, enough data) is better than small data because I can estimate what I want with sufficient precision. Its also worth noting that there is far far more important things than sample size which go into medical research. So the buck doesn't stop with saying you detected a large difference.

Now, I am not able to determine whether or not Dr. Raoul's statements re: Hydroxychloroquine are accurate. But, if his intention is to argue that differences detected in small groups are large (again, assuming differences are not false positives or the result of bias), then I can get behind that.

Answer 5

The quote in question seems to originate from marianne.net (in French) and, as it stands, is definitely wrong. But, as Demetri and Dave pointed out, with some language bending there might be some truth to it.

In my understanding, Prof. Raoult confuses significance and effect size. In a small sample, the effect size has to be large (i.e. of practical relevance) to be statistically significant. In large samples, even very small effects, negligible for all practical purposes, can be statistically "significant".

Just as a practical example: If the true effect of a drug is to prolong the life of a patient by, on average, one day, it is most likely useless for all practical purposes. In a small sample, say 20 persons, this small life extension will probably drown in the noise and wouldn't be noticeable at all. In a sample of $10^9$ persons, you might be able to see it.

That doesn't mean that smaller samples are better. Just because you have found that the effect is non-zero doesn't mean that the hypothetical drug is worth its price (I assume there are some direct cost associated with it, and there are probably other opportunity costs). "Statistical significance" is not the right criterion for making decisions, and even the effect size isn't enough (although you should always look at it). Decision making always involves balancing costs and benefits.

As of refuting the original statement:

If a smaller data set is better, why don't we take the empty set, of size zero, and simply announce the result which is the most convenient to us?

Answer 6

A smaller sample size is not better.

A small sample size needs a more significant* result if you want to draw a conclusion from it.

Let's consider some results and their interpretation:

• If your drug cures 30% of 10 people, the percentage of the general population cured could be anywhere between around 0% and 65% of people.

• If your drug cures 30% of 10000 people, you can be quite sure it actually cures around 30% of people (more specifically, between 29% and 31% of people).

• If your drug cures 100% of 10 people, you can be quite sure it would cure around at least 60% of people.

• If your drug cures 100% of 10000 people, you can be quite sure it actually cures around 100% of people.

Note: the above probably misses a few details about control groups, side effects, hypothesis testing, etc. It's just meant to give a basic idea of what the numbers might look like.

Now a one-line conclusion of a study could be "the drug likely cures some percentage of people" or "we don't know whether it cures anyone".

A 10000-person study is going to end up saying "the drug likely cures some percentage of people" more often, even if the percentage is really tiny. A 10-person study will end up saying "we don't know whether it cures anyone" more often.

When the 10-person study does end up saying "the drug likely cures some percentage of people", the percentage will generally be larger.

When a 10000-person study says "we don't know whether it cures anyone", we can be pretty sure that it cures between 0% and a very, very tiny percentage of the population. Whereas with a 10-person study with the same conclusion it could still cure a fairly large percentage. We just don't know yet.

But the results themselves are not more significant.

Note that above I didn't say "the results are more significant", but rather that you need more significant results. And I'm differentiating the results from the conclusion.

The quote (without context) seems to imply a smaller sample provides a more useful result, when this is blatantly false. This may not be what the author actually meant, but that's how I read it.

The results from a large study allows us to be more sure how effective something actually is, which is always more useful.

The only thing that would be more significant would be a positive conclusion ("the drug likely works"), but taking one look at the actual percentages would still give you a lot more information for the large study.

The only way in which a smaller sample would provide a more useful result is when people who don't know what they're doing misinterpret or misrepresent the result (by e.g. saying "the drug works" without also noting that it actually only works 1% of the time). This admittedly might happen a whole lot more often than it should in today's world with the media and social media.

What about bias?

If you have a very small sample size, you're much more likely to not have a sample that's proportional to what the actual population looks like, and you might even miss out on some demographic altogether.

In medicine there are many variables that could contribute to or alter the effects something has, so having an accurate representation of the population is quite important.

If your data is too biased, your results would not be particularly useful.

A bigger sample size doesn't automatically fix it, but does make it easier to avoid.

---

*: this answer uses "significant" to mean "practically significant" not "statistically significant". As in "something that actually matters to the general public".

Results from larger samples would generally be more statistically significant, as in it's something we can be more sure about.

Answer 7

There are a few things that are true, and worth understanding for where the confusion might slip in.

First, it is possible to get high levels of confidence from small samples, so long as the effect is sufficiently pronounced. For example, a treatment that goes from 10% control recovery to 90% experimental recovery will show up with a very good score even if you only have 20 samples. It will be better than a treatment going from 49.9% to 50.1% shown on a hundred samples. Of course, for the same treatment a bigger sample is still better, but a small sample may be enough.

Second, a small sample being enough is much more likely if the effect is more pronounced. This is one of these all-too-common effects in statistics that things flip around depending on whether you're talking about before or after the experiment. The spread from randomness is larger in both directions with smaller samples. Effectively, to plan a demonstration of your hypothesis with confidence, you need enough margin that even if randomness goes against you, and everyone presumes randomness went for you, you can still show an effect. To do that you need to reduce the effect of randomness, either by having a large sample, or by having a strong effect. So if you're planning your experiment and expect the effect to be very strong, then you can afford to use a smaller sample. (although you still shouldn't expect bonus points for doing so!) If you're planning an experiment and expect the effect to be subtle, then you'll need a much larger sample.

What this does not mean is that a small sample ever implies a more trustworthy result. To someone assessing the research, a 10% shift in outcome shown with a sample of 1000 is strictly better than a 10% shift in outcome with a sample of 20.

Strong effect implies small sample will (probably) be enough. Small sample does not imply a strong effect.

Answer 8

This statement is misleading because it is unclear what he means by significant.

In the case of a clinical trial, what you want to show is that people are more likely to heal when given a test treatment than when given a placebo. So you have two (random) groups of equal size, one of which gets a the treatment while the other gets a placebo. Then you observe the proportion of people who heal in each group, and this is where you must be careful what you call significant :

• Is there a large difference in the observed number of healing people in each group?(e.g. 3/4 of people healed in test group vs 1/2 in control group). This is purely based on one observation, and can be completely random due to natural variance (even with two placebo groups, one could by pure luck get more people healing than the other). For this definition of significant, smaller groups produce more significant results.

• Is the difference in observed healing rate (even if its absolute value is small) statistically significant ? This does not mean that there is a large difference between the two groups, but that we can know with high confidence that this difference is not due to random fluctuations. With a very large group, you can observe a very small difference in healing rate (e.g. 54% vs 55%) and still know it is not by chance. This is achieved using statistical techniques such as the Central Limit Theorem (CLT). For this definition of significant, larger groups produce more significant results.

• Is the difference large in absolute value given that the test is statistically significant? i.e. "Given that we know our results are not due to pure luck, are they of valuable practical use?". As some have already said, if you somehow manage to obtain a statistically significant result with a small sample, it is likely that your difference in absolute value is pretty large because small sample size wouldn't be able to detect a small difference in a statistically significant way. Also, while it is technically possible, be careful when checking statistical significance on small sample size because the usual asymptotic theorems such as the aforementioned CLT don't apply (I'm sure some happily use them anyway...). For this definition, smaller groups produce more significant but this is a case I wouldn't expect to encounter often, and I would be careful.

Hence, depending on which definition the author is using, he could be right or wrong. If he is using the first one, he is technically right but this number alone is useless in practice; if he's using the second one, he is simply wrong; and if he is using the third one, he is technically right but I still find it kind of suspicious for the reasons I mentioned.

Answer 9

A smaller sample size is definitely not better than a larger one.

Other answers do a good job of parsing what he might have meant (e.g., effect size, etc.). However, these miss the fact that the underlying effect is the same whether we use a larger or small sample size.

This is more obvious if we look at a binary outcome (e.g., survived vs died).

What happens if half of all patients die of the disease and the treatment has no effect? If we only sample three individuals in each group (treated and not), then 1.5% of the time, all three non-treated individuals will die and all three treated individuals will survive; an additional 4.5% of the time, all three non-treated individuals will die and two treated individuals will survive. So, about 6% of the time that you run this study, it would look like the treatment had a huge impact.

(6% of the time you would get the reverse result, but then the bias of the File Drawer Problem rears its head; p-hacking by adding samples for in-between results introduce yet another bias.)

That risk gets less severe as the sample size increases. Even with only 10 samples of each, the odds of seeing all non-treated die and all treated survive is only 0.0000954%.

The risk of a false-positive remains the same (because that is how p-values are defined), but we would have far better confidence in the estimate of the effect size (this is why confidence intervals shrink with increasing sample size). That confidence in the effect size is crucial, particularly for assessing the risk-reward tradeoff of a treatment with side-effects as severe as hydroxychloroquine and when rationing is already leading to problems from patients no longer able to access the drug for treatment of conditions that we know are mitigated by hydroxychloroquine (e.g., lupus).

Answer 10

Before really answering the question, I have to point out that the study being discussed was a non-randomized open label study where the controls were possibly in a different facility than the treatment subjects, and they tossed out dead, ICU patients and dropouts due to side effects from the treatment group. The poor outcomes (it appears) came from the treatment group, but they based the analysis on a surrogate endpoint - PCR detection of viruses. The use of a 6-day endpoint also seems post hoc. Thus I'd not be surprised if there were a LARGE amount of bias in the results.

So, the comments on sample size perhaps hold, but only if an incomplete analysis is performed. Usually, when you get statistically significant results (or even if you don't), you should follow up with confidence intervals for interesting effects. Dr. Raoult's argument is that tiny errors or biases in a study with huge sample sizes will cause the null hypothesis to be rejected despite a tiny effect size that could be due to poor procedures. In a small sample size, bias due to poor procedures is unlikely to cause the null to be rejected (if it were true), so a rejection would be associated with an apparent large estimated effect size.

This argument falls apart immediately when you follow up with a confidence interval. For the situation where there is a tiny bias and a large sample size (and the null is true), you'll get a narrow C.I. around a result that is apparently not practically significant. With the small sample size, you'll get a very wide confidence interval leaving little faith in the results at all.

If Dr. Raoult's thoughts on sample size were correct, then an equally good procedure would be to add a large amount of noise to your dataset to decrease the chance of accidental rejection of the null hypothesis in the presence of small experimental bias (a small sample size was advocated for just this reason). This doesn't appear especially wise.

Answer 11

First of I would like to state the following:

1. Biostatistics is a really difficult field; many biostatisticians are better data scientists than people with a maths background.
2. Biostatistics has create it own tools that we can use today.
3. The experiments they make must be really regulated (at least from pharmaceutical view)

Now imagine a trait that is really different between all of us. Let's experiment on height. That trait has indeed a high variance in the population. Will it follow a normal distribution? Of course yes we are talking about 7 billion people.

Now take 10 people from 10 different countries but you have to choose a country of at least one continent. You take the estimated world median and run a simple Mann-Whitney test to see if the estimated median is statistically different from your sample's median.

There are 2 possible results:

1. There is indeed a signifigant difference, so your stratified sample is not so effective
2. There is no difference whatsoever and the two medians seem to be the same with p-value=.0001

In the first case, there is noone that can argue with your experiment: it was wrong all along to test a sample with so few participants. In the second case there is a really good question to be answered: if you repeat your experiment about 20 times and have the exact same result; is this series of experiments better than an experiment with 10,000 participants? And if so, what does this means about our variable?

Well in the highly unlikely case of the 20 repeated experiments to be significant, it is really to scratch your head. We just used a good sampling technique and nothing fancy so we are aware of the whole procedure and we cannot see any bugs or shade areas and on the top of that the experiment had given us the statistically correct result!

But wait what if we had taken 10,000 people from different countries while having in mind the same sampling technique: stratified sampling. You ran the test again (we can still use Mann-Whitney although they should follow normal distribution and t-test is more powerful) and you see that the test says something unpredictable: the median of your sample is not the same as this of the population!

You can go home and be done with statistics in this hypothetical scenario, however there should be an explanation. Which is pretty simple. Variables with high variances have huge amount of outliers and many tests (like Mann-Whitney or t-test) are really susceptible to them. You have taken 10,000 people; how many of them do you expect to be an outlier?

Now consider a pharmaceutical experiment and the underlying variables (genes, environment, food etc etc). You should consider all these unique outliers that exists out there to have the best experiment, which is practically impossible.

So what do we gain from a smaller sample? We are granted with the holy grail of riskiness. I am not kidding. A smaller sample is likely to be so variant that there is no way to compare it with any distribution and sleep well at night. However if you can see a pattern in such a small (and always random with a really careful sampling method) sample there is really good potential to your theory.

So our problem is not wether or not the test is really significant or not (and I am starting to hearing your complaints but hear me out) is about how reliable is the sample. That's why your everyday painkiller has a telephone number on it. Because despite the large sample of the experiments anyone could still be in the outlier zone that the created model does not explain.

So the statement is really correct. The question is how good was their sampling method.

So what about his research? Well I read that it

does not meet the [International Society of Antimicrobial Chemotherapy’s] expected standard, especially relating to the lack of better explanations of the inclusion criteria and the triage of patients to ensure patient safety.

found here

so the problem of the International Society of Antimicrobial Chemotherapy’s was not the statistical method, rather the sampling method they used.

Answer 12

tl;dr– It sounds like they're arguing that smaller data sets are superior because larger data sets imply $p$-hacking or/and a less observable finding. But, obviously, doing a particular experiment with more data is better than with less data when the data analysis is done correctly.

---

Translating the argument.

1. Raw argument:

   It's counterintuitive, but the smaller the sample size of a clinical test, the more significant its results are. The differences in a sample of 20 people may be more significant than in a sample of 10,000 people. If we need such a sample, there is a risk of being wrong. With 10,000 people, when the differences are small, sometimes they don't exist.

2. Initial paraphrasing pass:

   The less data collected, the better the findings are. For example, findings based on 20 data points can be better than findings based on 10,000 data points. Because, if you get 10,000 data points, that implies that you couldn't find what you were looking for with just 20 data points. Even if you do eventually find something with such a large data set, it's probably going to be a smaller effect that might not even exist.

3. Rewriting the entire thing:

   Results are better when they're based on smaller data sets. The problem with large data sets is that they imply that the researchers failed to find the effect with a smaller data set, forcing them to resort to collecting more data. So, larger data sets imply weaker, less significant findings.

They seem to have two arguments in favor of smaller data sets:

1. Smaller data sets imply that the studied effect was more observable.

2. Larger data sets suggest that the researchers may've kept collecting data until they found the result they wanted, i.e. "optional stopping" as described in this question.

Of course, this argument completely fails to address the fact that, all else held equal, more data is superior to less. For example, even if they think that 20 data points was sufficient to find some effect, clearly 10,000 data points would be better.

Answer 13

The requirement that sample sizes be a certain size to have statistical inference confidence stems from I think the emperical law. And that is as you take more random SAMPLES the average of the MEANS converge to the actual population mean. But I've heard in order to be valid a sample size of greater than 32 is required for all samples. But there are other methods used for small sample sizes. You have to use the correct inferential statistics. But I do not know how sample size converge when the sample sizes are effected. I would think you need more data and there might be some notion of conservation relating error and number of samples, and sample size.

Edit after some simple algebra can see that if you have m samples of size n you should get same mean as one sample of size mn. Also it could be true for some random distribution that the error may oh increase its only when very large samples are taken that they have more likely hood of being inside the required intervals. So it seems it could depend actually on the distribution itself. But often you can't know that.

Also usually we use sample statistics to infer population statistics, not individual means or cases. Because then it is simple the probability described by the unknown population and we can only guess. Even if were right you can only know something to accuracy the probability distribution alows. But in statistics when we talk about statistics like population mean, we actually can get to the desired accuracy. Also from data perspective using samples we can throw away the data and store sample means.

But as for this question I think it is hard to say what this person actually means, it not simple and clear concrete statement. But if he is saying that smaller samples are more accurate, it could be plausible but if any statistics are of use then much larger samples would ultimately be best.

Answer 14

Dr. Raoult's statement is false.

A bigger dataset (on your domain of discourse subject) is always better. It yields a better average, thus better certainty.

You may apply the principle of charity.

Probably what he is trying to say is:

• A small (but most significant) sample set is better than a bigger (but less significant).

Imagine you are sampling hydrochloroquine efficacy in treating patients with Covid-19, but you sample randomly (without testing for Covid-19). This'll yield a misleading average.

From a scientific point of view, in general, you should ignore non-scientific press.

Answer 15

Yes, a smaller sample CAN be more informative due to the confluence of similar significant discriminating attributes. For example, a small sample of Detriot African Americans with similar income levels and associated access to healthcare, occupations, stress, diets and pre-existing conditions (high blood pressure,...) is better than a 100,000 sample of Americans where the confluence of these attributes is rare.

This is a problem in model design and conformity of the data to underlying assumptions of independence and identically distributed.

More is not always better, my field experience confirms 'garbage in' does result in 'garbage out', where increasing the sample size is not a solution nor is postulating explanatory variables after seeing the data.