Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Is there a "rule" to determine the minimum sample size required for a t-test to be valid?

For example, a comparison needs to be performed between the means of 2 populations.

There are 7 data points from one population and only 2 data points from the other.

Unfortunately, the experiment is very expensive and time consuming, and obtaining more data is not feasible.

Can a t-test be used? Why or why not? Please provide details (the population variances and distributions are not known).

If a t-test can not be used, can a non parametric test (Mann Whitney) be used? Why or why not?

Thanks a lot for your help! I know very little about stats, and have been searching around the internet for a detailed answer for some time.

share|improve this question
1  
This question covers similar material & will be of interest to viewers of this page: Is there a minimum sample size required for the t-test to be valid?. –  gung Oct 19 '12 at 22:34
add comment

7 Answers 7

up vote 4 down vote accepted

I'd recommend using the non-parametric Mann-Whitney U test rather than an unpaired t-test here.

There's no absolute minimum sample size for the t-test, but as the sample sizes get smaller, the test becomes more sensitive to the assumption that both samples are drawn from populations with a normal distribution. With samples this small, especially with one sample of only two, you'd need to be very sure that the population distributions were normal -- and that has to be based on external knowledge, as such small samples gives very little information in themselves about the normality or otherwise of their distributions. But you say that "the population variances and distributions are not known" (my italics).

The Mann-Whitney U test does not require any assumptions about the parametric form of the distributions, requiring only the assumption that the distributions of the two groups are the same under the null hypothesis.

share|improve this answer
2  
Not a good recommendation for extremely small sample sizes. With 7 and 2 samples, the U-test will fail, no matter how large the difference between the mean of the groups. Look at my answer for an example. –  AlefSin Jan 13 at 11:14
    
I would second what @AlefSin says. If it is important for you to draw valid conclusions (and not only get a p-value) then the more resonable assumptions you can make the better. If there is reasonable background information you could also add even more assumptions if you did your analysis in a Bayesian framework. –  Rasmus Bååth Jan 29 at 16:09
    
One problem is that with such small sample sizes, the Wilcoxon-Mann-Whitney cannot achieve typical significance levels. With sample sizes of 7 and 2 you'll never get a result significant at the 5% level, no matter how blatant the difference. Consider (1.018,1.031,1.027,1.020,1.021,1.019,1.024) vs (713.2, 714.5) -- not significant at the 5% level! –  Glen_b Mar 30 at 23:59
add comment

There is no minimum sample size for a t-test; the t-test was, in fact, designed for small samples. In the old days when tables were printed, you saw t-test tables for very small samples (as measured by df).

Of course, as with other tests, if there is a small sample only quite a large effect will be statistically significant.

share|improve this answer
    
But will the opposite also be a problem? I.e., might outliers that happened to be sampled lead to falsely rejecting the null hypothesis? Or is low power to detect differences a bigger problem? In this particular situation I am seeing a significant difference between the means but don't know how much to "trust" it. –  Johnny Puzzled Jan 21 '12 at 23:09
1  
With n=2 you are definitely vulnerable to the influence of outliers--outliers in the population; how can a sample of 2 have an outlier within the sample? :-) I wouldn't try any inferential statistics in this situation. Prospects are poor for getting at the "truth," and you'll be leaving yourself wide open to criticism. –  rolando2 Jan 22 '12 at 0:23
1  
The reason that the confidence interval will be wide is precisely because you might get an outlier. But t-test still assumes samples are from a normal population. –  Peter Flom Jan 22 '12 at 1:54
add comment

I assume you mean you have 7 data points from one group, and 2 data points from a second group, both of which are subsets of populations (e.g. subset of males and subset of females).

The maths for the t-test can be obtained from this Wikipedia page. We will assume an independent two-sample t-test, with unequal sample sizes (7 vs. 2) and unequal variances, so about half-way down that page. You can see that the calculation is based on means and standard deviations. With only 7 subjects in one group and 2 subjects in another, you cannot assume you have good estimates for either the mean or the standard deviation. For the group with 2 subjects, the mean is simply the value that lies exactly in the middle of the two data points, so it is not well estimated. For the group with 7 subjects, sample size strongly affects variances (and therefore standard deviations, which are the square root of the variance) because extreme values exert a much stronger effect when you have a smaller sample.

For example, if you look at the basic example on the Wikipedia page for standard deviation you will see that the standard deviation is 2, and the variance (square the standard deviation) is therefore 4. But if we only had the first two data points (the 9 and the 1), the variance would be 10/2 = 5 and the standard deviation would be 2.2 and if we only had the last two values (the 4 and the 16), the variance would be 20/2 = 10 and the standard deviation would be 3.2. We're still using the same values, just less of them, and we can see the effect on our estimates.

That is the problem with using inferential statistics with small sample sizes, your results will be particularly strongly affected by sampling.

Update: is there any reason why you can't simply report the results by subject and indicate that this is exploratory work? With only two cases, the data is very similar to a case study, and these are both (1) important to write up and (2) accepted practice.

share|improve this answer
    
Thanks Michelle. This is interesting and useful to know. However, what would you recommend from a practical point of view? Given this situation, what is the best way to proceed? Thanks! –  Johnny Puzzled Jan 21 '12 at 23:05
    
Hi Johnny Puzzled. Without more information on your exact situation I feel unable to give more guidance. –  Michelle Jan 21 '12 at 23:35
    
What kind of information is needed? –  Johnny Puzzled Jan 21 '12 at 23:44
1  
Hi again, more information on your study design, such as what your data is, how you collected it, what your groups are, how the observations were selected. All I know is that you did an experiment with 9 observations (people? rats? neurons? blocks of cheese? radiation frequencies?) that are from two groups. –  Michelle Jan 21 '12 at 23:59
    
Let's say that the average blood flow to white matter in the brain was measured in humans using MRI. The groups are controls (7 people) and age/sex matched patients with a particular disorder (2 people). –  Johnny Puzzled Jan 22 '12 at 12:35
add comment

(disclaimer: I cannot type well today: my right hand is fractured!)

Contrary to the advice to use a non-parametric test in other answers, you should consider that for extremely small sample sizes those methods are not very useful. It is easy to understand why: in studies with extremely small size, no difference between groups can be established unless a big effect size if observed. Non-parametric methods, however, do not care for the magnitude of the difference between the groups. Thus even if the difference between the two groups is huge, with a tiny sample size a non-parametric test will always fail to reject the null hypothesis.

Consider this example: two groups, normal distribution, same variance. Group 1: average 1.0, 7 samples. Group 2: average 5, 2 samples. There is a big difference between the averages.

wilcox.test(rnorm(7, 1), rnorm(2, 5))

   Wilcoxon rank sum test

data:  rnorm(7, 1) and rnorm(2, 5)
W = 0, p-value = 0.05556

The computed p-value is 0.05556 which does not reject the null hypothesis (at 0.05). Now, even if you increase the distance between the two means by a factor of 10, you will get the same p-value:

wilcox.test(rnorm(7, 1), rnorm(2, 50))

   Wilcoxon rank sum test

data:  rnorm(7, 1) and rnorm(2, 50)
W = 0, p-value = 0.05556

Now I invite you to repeat the same simulation with t-test and observe the p-values in the case of large (average 5 vs 1) and huge (average 50 vs 1) differences.

share|improve this answer
add comment

Interesting related article: 'Using the Student's t-test with extremely low samlpe sizes' J.C.F de Winter (in Practical Assesment, Research & Evaluation) http://goo.gl/ZAUmGW

share|improve this answer
add comment

I would recommend to compare the conclusions that you get with both, the t-test and the Mann-Whitney test, and also take a look at boxplots and the profile likelihood of the mean of each population.

share|improve this answer
    
Hi @Demian, I'm not sure that even a boxplot will be helpful when one group has a sample size of 2. Otherwise, yes I think boxplots in particular are very helpful in visualising continuous data across groups. –  Michelle Jan 22 '12 at 22:54
add comment

Stata 13/SE code for a bootstrap ttestAs a ttest performed on small samples probably does not fulfill the ttest requirements (mainly, the normality of the populations from which the two samples have bee drawn), I would recommend to perform a bootstrap ttest (with unequal variances), following Efron B, Tibshirani Rj. An Introdution to the Bootstrap. Boca Raton, FL: Chapman & Hall/CRC, 1993: 220-224. The code for a bootstrap ttest on the data provided by Johnny Puzzled in Stata 13/SE is reported in the image above.

share|improve this answer
    
Your answer has serious formatting issues, would you mind editing it? –  amoeba Jan 29 at 10:07
    
I have tried to solve formatting issues in the reviewed version of the answer. Thanks to amoeba for pointing this out. –  Carlo Lazzaro Jan 29 at 16:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.