Can I trust a significant result of a t-test if the sample size is small?

Question

If my one sided t-test result is significant but the sample size is small (e.g. below 20 or so), can I still trust this result? If not, how should I deal and/or interpret this result?

Answer 1

In theory if all the assumptions of the t-test are true then there's no problem with a small sample size.

In practice there are some not-quite-true assumptions which we can get away with for large sample sizes but they can cause problems for small sample sizes. Do you know if the underlying distribution is normally distributed? Are all the samples independent and identically distributed?

If you doubt the validity of the test then an alternative you can make use of is bootstrapping. Bootstrapping involves resampling from your sample in order to see how often the null hypothesis is true or false. Perhaps your null hypothesis is $\mu<0$ and your p-value is 0.05 but the bootstrapping shows that the sample mean is less than zero 10% of the time. This would indicate that it was a fluke which caused a p-value of 0.05 and you should be less confident that the null hypothesis is false.

Answer 2

You should rarely trust any single significant result. You didn't say why you were using a one-tailed instead of a two-tailed test, so hopefully you have a good reason for doing so other than struggling to be able to claim a statistically significant outcome!

Setting that aside, consider the following from p. 261 of Sauro, J., & Lewis, J. R. (2016). Quantifying the User Experience: Practical Statistics for User Research, 2nd Ed.. Cambridge, MA: Morgan-Kaufmann.

---

How Ronald Fisher recommended using p-values

When Karl Pearson was the grand old man of statistics and Ronald Fisher was a relative newcomer, Pearson, apparently threatened by Fisher’s ideas and mathematical ability, used his influence to prevent Fisher from publishing in the major statistical journals of the time, Biometrika and the Journal of the Royal Statistical Society. Consequently, Fisher published his ideas in a variety of other venues such as agricultural and meteorological journals, including several papers for the Proceedings of the Society for Psychical Research. It was in one of the papers for this latter journal that he mentioned the convention of setting what we now call the acceptable Type I error (alpha) to 0.05 and, critically, also mentioned the importance of reproducibility when encountering an unexpected significant result:

An observation is judged to be significant, if it would rarely have been produced, in the absence of a real cause of the kind we are seeking. It is a common practice to judge a result significant, if it is of such a magnitude that it would have been produced by chance not more frequently than once in twenty trials. This is an arbitrary, but convenient, level of significance for the practical investigator, but it does not mean that he allows himself to be deceived once in every twenty experiments. The test of significance only tells him what to ignore, namely, all experiments in which significant results are not obtained. He should only claim that a phenomenon is experimentally demonstrable when he knows how to design an experiment so that it will rarely fail to give a significant result. Consequently, isolated significant results which he does not know how to reproduce are left in suspense pending further investigation. (Fisher, 1929, p. 191)

Reference

Fisher, R. A. (1929). The statistical method in psychical research. Proceedings of the Society for Psychical Research, 39, 189-192.

Answer 3

Imagine yourself to be in a situation where you're doing many similar tests, in a set of circumstances where some fraction of the nulls are true.

Indeed, let's model it using a super-simple urn-type model; in the urn, there are numbered balls each corresponding to an experiment you might choose to do, some of which have the null true and some which have the null false. Call the proportion of true nulls in the urn $t$.

To further simplify the idea, let us assume the power for those false nulls is constant (at $(1-\beta)$, since $\beta$ is the usual symbol for the type II error rate).

You choose some experiments from our urn ($n$ of them, say) "at random", perform them and reject or fail to reject their hypothesis. We can assume that the total number of experiments in the urn ($M$, say) is large enough that it doesn't make a difference that this is sampling without replacement (i.e. we'd be happy to approximate this as a binomial if need be), and both $n$ and $M$ are large enough that we can discuss what happens on average as if they're what we experience.

What proportion of your rejections will be "correct"?

Expected total number of rejections: $nt\alpha+n(1-t)(1-\beta)$
Expected total number of correct rejections: $n(1-t)(1-\beta)$

Overall proportion of times a rejection was actually the right decision: $\frac{(1-t)(1-\beta)}{t\alpha+(1-t)(1-\beta)}$

Overall proportion of times a rejection was an error: $\frac{t\alpha}{t\alpha+(1-t)(1-\beta)}$

For the proportion of correct rejections to be more than a small number you need to avoid the situation where $(1-t)(1-\beta)\ll t\alpha$

Since in our setup a substantial fraction of nulls are true, if $1-\beta$ is not substantially larger than $\alpha$ (i.e. if you don't have fairly high power), a lot of our rejections are mistakes!

So when your sample size is small (and hence power is low), if a reasonable fraction of our nulls were true, we'd often be making an error when we reject.

The situation isn't much better if almost all our nulls are strictly false -- while most of our rejections will be correct (trivially, since tiny effects are still strictly false), if the power isn't high, a substantial fraction of those rejections will be "in the wrong direction" - we'll conclude the null is false quite often because by chance the sample turned out to be on the wrong side (this may be one argument to use one sided tests - when one sided tests make sense - to at least avoid rejections that make no sense if large sample sizes are hard to get).

We can see that small sample sizes can certainly be a problem.

[This proportion of incorrect rejections is called the false discovery rate]

---

If you have a notion of likely effect size you're in a better position to judge what an adequate sample size might be. With large anticipated effects, a rejection with a small sample size would not necessarily be a major concern.

Answer 4

Some of Gosset's original work (aka Student), for which he developed the t test, involved yeast samples of n=4 and 5. The test was specifically designed for very small samples. Otherwise, the normal approximation would be fine. That said, Gosset was doing very careful, controlled experiments on data that he understood very well. There's a limit to the number of things a brewery has to test, and Gosset spent his working life at Guinness. He knew his data.

I'm a bit suspicious of your emphasis on one-sided testing. The logic of testing is the same whatever the hypothesis, but I've seen people go with a significant one-sided test when the two-sided was non-significant.

This is what a (upper) one-sided test implies. You are testing that a mean is 0. You do the math and are prepared to reject when T > 2.5. You run your experiment and observe that T=-50,000. You say, "phhhhht", and life goes on. Unless it is physically impossible for the test statistic to sink way below the hypothesized parameter value, and unless you would never take any decision if the test statistic goes in the opposite direction than you expect, you should be using a two-sided test.

Answer 5

The main thing you need to worry about is the power of your test. In particular, you might want to do a post-hoc power analysis to determine how likely you are, given your sample size, to identify a true significant effect of a reasonable size. If typical effects are very large, an n of 8 could be totally adequate (as with many experiments in molecular biology). If the effects you are interested in are typically subtle, however (as in many social psychology experiments), an n of thousands might still be underpowered.

This is important because underpowered tests can give very misleading results. For example, if your test is underpowered, even if you find a significant result, you have a relatively high probability of making what Andrew Gelman calls a "Type S" error, i.e., there is a real effect but in the opposite direction, or a "Type M" error, i.e., there is a real effect but the true magnitude is much weaker than what is estimated from the data.

Gelman and Carlin wrote a useful paper about doing post-hoc power analysis that I think applies in your case. Importantly, they recommend using independent data (i.e., not the data you tested, but reviews, modeling, the results of similar experiments, etc.) to estimate a plausible true effect size. By performing power analysis using that plausible estimated true effect size and comparing to your results, you can determine the probability of making a Type S error and the typical "exaggeration ratio," and thus get a better sense for how strong your evidence really is.

Answer 6

One could say that the whole point of statistical significance is to answer the question "can I trust this result, given the sample size?". In other words, the whole point is to control for the fact that with small sample sizes, you can get flukes, when no real effect exists. The statistical significance, that is to say the p-value, is precisely the answer to the question, "if no real effect existed, how likely would I be to get a fluke as big as this?". If it's very unlikely, that indicates that it's not a fluke.

So the answer is "yes", if the p-value is low, and if you have followed the correct statistical procedures and are satisfying the relevant assumptions, then yes, it is good evidence, and has the same weight as if you'd gotten the same p-value with a very large sample size.