A significant result from an experiment with a small sample size

Question

When a study with a small sample size finds a significant result in null hypothesis significance testing, what is the significant concern about it (e.g., reviewers can complain about the sample size even when results are significant)? If a study does not find a significant result, I can see that it (i.e., not rejecting the null hypothesis) does not mean much (e.g., it may just mean a low power). But if it has a significant result, the power argument cannot be used.

I can think that small sample size is not enough to check the assumptions of the hypothesis test (e.g., distribution assumptions for parametric tests) that are being used. At the same time, extremely large sample size will always find such assumptions to be violated (e.g., it is impractical to think any natural distributions are exactly normal distributions or other assumed distributions).

Answer 1

In my opinion, one should avoid dichotomizing results in "significant"/"not-significant" altogether.

If you report the effect size (e.g., the difference between means), a plot of the data, and the pvalue for the null hypothesis, then you can discuss their interpretation and let the reader make up their mind. With lots of data points the pvalue probably becomes meaningless, but that also means you can focus on the effect size and its interpretation.

Of course, in many real-world cases you need to dichotomize, e.g. to decide to follow up a project or not based on preliminary results. But if the sample size is small, that decision is going to be tricky whatever statistics you use.

Answer 2

You’ve run into one of the arguments against NHST. With large enough sample size even trivial differences may attain traditional significance levels. But that’s beside the point.

With small sample sizes the concern is that you are capitalizing on chance... that you have a few data points that are not representative of the population that carry the effect.

Answer 3

I would like to add, that the power argument can still be used if you have a significant result.

Think about $200$ small studies which are underpowered due to their small sample size, e.g. the power in all of these studies is $40\%$. Let's further say that one half of these studies has bad luck, and actually the null hypothesis is true, i.e. there is no effect in reality. The other half is lucky and the alternative hypothesis is actually true. Now, using the classical threshold for significance $\alpha=0.05$, among the $100$ studies for which the null hypothesis is true, there will be $5\%$ of them which have a significant p-value, so we expect $5$ studies with significant finding (and only those get published in this simplified world). For the other $100$ studies, where the alternative hypothesis is true, we will expect $40$ significant findings. Hence, we have $45$ significant findings in total, from which $5$ are false-positives, so there is $5/45=11.11\%$ probability that a significant finding is a false-positive.

Now let's repeat the same thought experiment for the scenario of a world with only well-powered studies, e.g. power is $80\%$. From the studies with bad luck, there will be again $5\%$ that have a significant finding. But from the studies with an actual effect, now $80\%$ will have a significant finding, so we expect $80$ studies with significant p-value. Which makes it $5/85=5.88\%$ false-positives.

In conclusion: significant results of poorly powered studies are less reliable. However, the whole argument concerns sample size only indirectly, because sample size is related to power. A study can be well powered even with extremely small samples, and then I do not see any issue. To be valid, the full sample size argumentation just needs to be stated clearly before conduct of the study.

But in practice (at least from what I see daily), many small studies did not do a proper sample size calculation. To be honest, large studies don't do so much better. And many studies perform extensive multiple testing which increases type I error to a large extent, so the numbers that I have used in the example above are probably even more extreme in reality.