# Minimum sample size for unpaired t-test

Is there a "rule" to determine the minimum sample size required for a t-test to be valid?

For example, a comparison needs to be performed between the means of 2 populations. There are 7 data points from one population and only 2 data points from the other. Unfortunately, the experiment is very expensive and time consuming, and obtaining more data is not feasible.

Can a t-test be used? Why or why not? Please provide details (the population variances and distributions are not known). If a t-test can not be used, can a non parametric test (Mann Whitney) be used? Why or why not?

-
This question covers similar material & will be of interest to viewers of this page: Is there a minimum sample size required for the t-test to be valid?. – gung Oct 19 '12 at 22:34
See also this question where testing with even smaller sample sizes is discussed. – Glen_b Apr 2 at 5:51

I'd recommend using the non-parametric Mann-Whitney U test rather than an unpaired t-test here.

There's no absolute minimum sample size for the t-test, but as the sample sizes get smaller, the test becomes more sensitive to the assumption that both samples are drawn from populations with a normal distribution. With samples this small, especially with one sample of only two, you'd need to be very sure that the population distributions were normal -- and that has to be based on external knowledge, as such small samples gives very little information in themselves about the normality or otherwise of their distributions. But you say that "the population variances and distributions are not known" (my italics).

The Mann-Whitney U test does not require any assumptions about the parametric form of the distributions, requiring only the assumption that the distributions of the two groups are the same under the null hypothesis.

-
Not a good recommendation for extremely small sample sizes. With 7 and 2 samples, the U-test will fail, no matter how large the difference between the mean of the groups. Look at my answer for an example. – AlefSin Jan 13 '14 at 11:14
I would second what @AlefSin says. If it is important for you to draw valid conclusions (and not only get a p-value) then the more resonable assumptions you can make the better. If there is reasonable background information you could also add even more assumptions if you did your analysis in a Bayesian framework. – Rasmus Bååth Jan 29 '14 at 16:09
One problem is that with such small sample sizes, the Wilcoxon-Mann-Whitney cannot achieve typical significance levels. With sample sizes of 7 and 2 you'll never get a result significant at the 5% level, no matter how blatant the difference. Consider (1.018,1.031,1.027,1.020,1.021,1.019,1.024) vs (713.2, 714.5) -- not significant at the 5% level! – Glen_b Mar 30 '14 at 23:59
That said, if one has $n_1=7$ and $n_2=2$, there's a very good argument that one should perhaps consider whether a 5% test makes sense in the first place. A proper assessment of the cost of the two error types may lead to quite a different choice. – Glen_b Apr 2 at 5:53

There is no minimum sample size for a t-test; the t-test was, in fact, designed for small samples. In the old days when tables were printed, you saw t-test tables for very small samples (as measured by df).

Of course, as with other tests, if there is a small sample only quite a large effect will be statistically significant.

-
But will the opposite also be a problem? I.e., might outliers that happened to be sampled lead to falsely rejecting the null hypothesis? Or is low power to detect differences a bigger problem? In this particular situation I am seeing a significant difference between the means but don't know how much to "trust" it. – Johnny Puzzled Jan 21 '12 at 23:09
With n=2 you are definitely vulnerable to the influence of outliers--outliers in the population; how can a sample of 2 have an outlier within the sample? :-) I wouldn't try any inferential statistics in this situation. Prospects are poor for getting at the "truth," and you'll be leaving yourself wide open to criticism. – rolando2 Jan 22 '12 at 0:23
The reason that the confidence interval will be wide is precisely because you might get an outlier. But t-test still assumes samples are from a normal population. – Peter Flom Jan 22 '12 at 1:54

(disclaimer: I cannot type well today: my right hand is fractured!)

Contrary to the advice to use a non-parametric test in other answers, you should consider that for extremely small sample sizes those methods are not very useful. It is easy to understand why: in studies with extremely small size, no difference between groups can be established unless a big effect size if observed. Non-parametric methods, however, do not care for the magnitude of the difference between the groups. Thus even if the difference between the two groups is huge, with a tiny sample size a non-parametric test will always fail to reject the null hypothesis.

Consider this example: two groups, normal distribution, same variance. Group 1: average 1.0, 7 samples. Group 2: average 5, 2 samples. There is a big difference between the averages.

wilcox.test(rnorm(7, 1), rnorm(2, 5))

Wilcoxon rank sum test

data:  rnorm(7, 1) and rnorm(2, 5)
W = 0, p-value = 0.05556


The computed p-value is 0.05556 which does not reject the null hypothesis (at 0.05). Now, even if you increase the distance between the two means by a factor of 10, you will get the same p-value:

wilcox.test(rnorm(7, 1), rnorm(2, 50))

Wilcoxon rank sum test

data:  rnorm(7, 1) and rnorm(2, 50)
W = 0, p-value = 0.05556


Now I invite you to repeat the same simulation with t-test and observe the p-values in the case of large (average 5 vs 1) and huge (average 50 vs 1) differences.

-

I assume you mean you have 7 data points from one group, and 2 data points from a second group, both of which are subsets of populations (e.g. subset of males and subset of females).

The maths for the t-test can be obtained from this Wikipedia page. We will assume an independent two-sample t-test, with unequal sample sizes (7 vs. 2) and unequal variances, so about half-way down that page. You can see that the calculation is based on means and standard deviations. With only 7 subjects in one group and 2 subjects in another, you cannot assume you have good estimates for either the mean or the standard deviation. For the group with 2 subjects, the mean is simply the value that lies exactly in the middle of the two data points, so it is not well estimated. For the group with 7 subjects, sample size strongly affects variances (and therefore standard deviations, which are the square root of the variance) because extreme values exert a much stronger effect when you have a smaller sample.

For example, if you look at the basic example on the Wikipedia page for standard deviation you will see that the standard deviation is 2, and the variance (square the standard deviation) is therefore 4. But if we only had the first two data points (the 9 and the 1), the variance would be 10/2 = 5 and the standard deviation would be 2.2 and if we only had the last two values (the 4 and the 16), the variance would be 20/2 = 10 and the standard deviation would be 3.2. We're still using the same values, just less of them, and we can see the effect on our estimates.

That is the problem with using inferential statistics with small sample sizes, your results will be particularly strongly affected by sampling.

Update: is there any reason why you can't simply report the results by subject and indicate that this is exploratory work? With only two cases, the data is very similar to a case study, and these are both (1) important to write up and (2) accepted practice.

-
Thanks Michelle. This is interesting and useful to know. However, what would you recommend from a practical point of view? Given this situation, what is the best way to proceed? Thanks! – Johnny Puzzled Jan 21 '12 at 23:05
Hi Johnny Puzzled. Without more information on your exact situation I feel unable to give more guidance. – Michelle Jan 21 '12 at 23:35
What kind of information is needed? – Johnny Puzzled Jan 21 '12 at 23:44
Hi again, more information on your study design, such as what your data is, how you collected it, what your groups are, how the observations were selected. All I know is that you did an experiment with 9 observations (people? rats? neurons? blocks of cheese? radiation frequencies?) that are from two groups. – Michelle Jan 21 '12 at 23:59
Let's say that the average blood flow to white matter in the brain was measured in humans using MRI. The groups are controls (7 people) and age/sex matched patients with a particular disorder (2 people). – Johnny Puzzled Jan 22 '12 at 12:35

Interesting related article: 'Using the Student's t-test with extremely low samlpe sizes' J.C.F de Winter (in Practical Assesment, Research & Evaluation) http://goo.gl/ZAUmGW

-

I would recommend to compare the conclusions that you get with both, the t-test and the Mann-Whitney test, and also take a look at boxplots and the profile likelihood of the mean of each population.

-
Hi @Demian, I'm not sure that even a boxplot will be helpful when one group has a sample size of 2. Otherwise, yes I think boxplots in particular are very helpful in visualising continuous data across groups. – Michelle Jan 22 '12 at 22:54

As a ttest performed on small samples probably does not fulfill the ttest requirements (mainly, the normality of the populations from which the two samples have bee drawn), I would recommend to perform a bootstrap ttest (with unequal variances), following Efron B, Tibshirani Rj. An Introdution to the Bootstrap. Boca Raton, FL: Chapman & Hall/CRC, 1993: 220-224. The code for a bootstrap ttest on the data provided by Johnny Puzzled in Stata 13/SE is reported in the image above.

-
Your answer has serious formatting issues, would you mind editing it? – amoeba Jan 29 '14 at 10:07
I have tried to solve formatting issues in the reviewed version of the answer. Thanks to amoeba for pointing this out. – Carlo Lazzaro Jan 29 '14 at 16:01

With a sample size of 2, the best thing to do may be to look at the individual numbers themselves and not even bother with statistical analysis.

-
At present this reads more like a comment. While this is a good point, for a reasonable answer to the original problem, some discussion of the issue itself might be expected, even if ultimately one concludes that it makes more sense to do something else. – Glen_b Apr 2 at 5:55