Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm currently working on a quasi-experimental research paper. I only have a sample size of 15 due to low population within the chosen area and that only 15 fit my criteria. Is 15 the minimum sample size to compute for t-test and F-test? If so, where can I get an article or book to support this small sample size?

This paper was already defended last Monday and one of the panel asked to have a supporting reference because my sample size is too low. He said it should've been at least 40 respondents.

share|improve this question
A sample size can be substantially smaller than 15 if the assumptions hold. Was the validity of the t-distribution the only reason he suggested a larger sample? – Glen_b Sep 26 '12 at 3:20
Just to clarify, what kind of t-test are you performing: one sample, paired sample or two sample. – Jeromy Anglim Sep 26 '12 at 4:23
Historically, the very first demonstration of the t-test (in "Student"'s 1908 paper) was in an application to sample sizes of size four. Indeed, obtaining improved results for small samples is the test's claim to fame: once the sample size reaches 40 or so, the t-test is not substantially different from the z-tests researchers had been applying throughout the 19th century. You may share a modern version of this paper with the panel member: Point out the investigation in Section VI, pp 14-18. – whuber Sep 26 '12 at 20:58
But you should ponder the fact that small sample sizes such as 4 works because Student had high-quality data: chemical lab data, experiments, not quasi-experiments. Your main problem is not with sample size but with representativity: How do you know that your data are representative of anything? – kjetil b halvorsen Sep 27 '12 at 7:08
@CzarinaFrancoise Why would we limit ourselves science <10 years old? – RioRaider Oct 6 '12 at 3:01

There is no minimum sample size for the t test to be valid. Validity requires that the assumptions for the test statistic hold approximately. Those assumptions are in the one sample case that the data are iid normal (or approximately normal) with mean 0 under the null hypothesis and a variance that is unknown but estimated from the sample. In the two sample case it is that both samples are independent of each other and each sample consists of iid normal variables with the two samples having the same mean and a common unknown variance under the null hypothesis. A pooled estimate of variance is used for the statistic.

In the one sample case the distribution under the null hypothesis is a central t with n-1 degrees of freedom. In the two sample cases with sample sizes n and m not necessarily equal the null distribution of the test staistics is t with n+m-2 degrees of freedom. The increased variability due to low sample size if accounted for in the distribution which has heavier tails when the degrees of freedom is low which corresponds to a low sample size. So critical values can be found for the test statistic to have a given significance level for any sample size (well at least of size 2 or larger).

The problem with low sample size is with regard to the power of the test. The reviewer may have felt that 15 per group was not a large enough sample size to have high power of detecting a meaningful difference say delta between the two means or a mean greater than delta in absolute value for a one sample problem. Needing 40 would require a specification of a certain power at a particular delta that would be achieved with n equal 40 but not lower than 40.

share|improve this answer

With all deference to him, he doesn't know what he's talking about. The t-test was designed for working with small samples. There isn't really a minimum (maybe you could say a minimum of 3 for a one-sample t-test, IDK), but you do have a concern regarding adequate power with small samples. You may be interested in reading about the ideas behind compromise power analysis when the possible sample size is highly restricted, as in your case.

As for a reference that proves you can use the t-test with small samples, I don't know of one, and I doubt that one exists. Why would anyone try to prove that? The idea is just silly.

share|improve this answer
+1 (to you and Michael). Of interest, you don't even need two observations to make inferences if willing to make a set of assumptions! – Andy W Sep 26 '12 at 0:15
The reason for the t test in small sample is that even when the samples are normal if the standard deviation is unknown the common thing to do is normalize by dividing by a sample estimate of the standard deviation. In large samples that estimate will be close enough to the population standard deviation that the test statistic will be approximately standard normal but in small sample it will have heavier tails then the normal. – Michael Chernick Sep 26 '12 at 0:19
The t distribution with n-1 degrees of freedom is the exact distribution for any sample size n under the null hypothesis and in small samples it need to be used in place of the normal which does not approximate it well. The real issue with sample size as both gung and I stated is power. If you want to argue with the referee that 15 is enough you need to identify how large a difference is needed to be called meaningful (the delta I mentioned) and then for that delta you need to show that the power is adequate say 0.80 or higher. – Michael Chernick Sep 26 '12 at 0:21
@CzarinaFrancoise About n>=30, see… – Stéphane Laurent Sep 26 '12 at 8:32
@gung Student's original (1908!) paper proves you can use the t-test with small samples. (For more about this, please refer to my extended comment to the original question.) – whuber Sep 26 '12 at 21:02

As mentioned in existing answers, the main issue with a small sample size is low statistical power. There are various rules of thumb regarding what is acceptable statistical power. Some people say 80% statistical power is reasonable, but ultimately, more is better. There is also generally a trade-off between the cost of getting more participants and the benefit of getting more statistical power.

You can assess statistical power of a t test using a simple function in R, power.t.test.

The following code provides the statistical power for a sample size of 15, a one-sample t-test, standard $\alpha=.05$, and three different effect sizes of .2, .5, .8 which have sometimes been referred to as small, medium, and large effects respectively.

p.2 <-power.t.test(n=15, delta=.2, sd=1, sig.level=.05, type='one.sample')
p.5 <- power.t.test(n=15, delta=.5, sd=1, sig.level=.05, type='one.sample')
p.8 <-power.t.test(n=15, delta=.8, sd=1, sig.level=.05, type='one.sample')

round(rbind(p.2=p.2$power, p.5=p.5$power, p.8=p.8$power), 2)  

p.2 0.11
p.5 0.44
p.8 0.82

Thus, we can see that if the population effect size was "small" or "medium", you would have low statistical power (i.e., 11% and 44% respectively). However, if the effect size is large in the population, you would have what some would describe as "reasonable" power (i.e., 82%).

The Quick-r website provides further information on power analysis using R.

share|improve this answer
Nice answer! Also there is a good software for computing statistical power called G*Power. – Enrique Jul 26 '15 at 18:23

The two-sample t-test is valid if the two samples are independent simple random samples from Normal distributions with the same variance and each of the sample sizes is at least two (so that the population variance can be estimated.) Considerations of power are irrelevant to the question of the validity of the test. Depending upon the size of the effect that one wishes to detect, a small sample size may be imprudent, but a small sample size does not invalidate the test. Note also that for any sample size, the sampling distribution of the mean is Normal if the parent distribution is Normal. Of course,larger sample sizes are always better because they provide more precise estimates of parameters. The Central Limit Theorem tells us that sample means are more Normally distributed than individual values, but as pointed out by Casella and Berger, it is of limited usefulness since the rate of approach to Normality must be checked for any particular case. Relying on rules of thumb is unwise. See the results reported Rand Wilcox's books.

share|improve this answer

While it is true that the t-distribution takes into account the small sample size, I would assume that your referee was thinking about the difficulty of establishing that the population is normally distributed, when the only information you have is a relatively small sample? This may not be a huge issue with a sample of size 15, since the sample hopefully is large enough to show some signs of being vaguely normally distributed? If this is true, then hopefully the population is somewhere near normal too and, combined with Central Limit Theorem, that ought to give you sample means that are well enough behaved.

But I'm dubious about recommendations to use t-tests for tiny samples (such as size four) unless the normality of the population can be established by some external information or mechanical understanding? There cannot surely be anywhere near enough information in a sample of size four to have any clue as the shape of the population distribution.

share|improve this answer

As far as assumptions go for the two sample case; it is that both samples are independent of each other and each sample consists of iid normal variables with the two samples having the same mean and a common unknown variance under the null hypothesis.

There is also the Welch t-test utilizing the Satterwaite Approximation for the standard error. This is a 2 sample t-test assuming unequal variances.'s_t_test

share|improve this answer

Czarina may find interesting to compare the results of her parametric t-test with the results obtained by a bootstrap t-test. The following code for Stata 13/1 mimics a fictitious example concerning a two-sample t-test with unequal variances (parametric t-test: p-value = 0.1493; bootstrap t-test: p-value = 0.1543).

set obs 15
g A=2*runiform()
g B=2.5*runiform()
ttest A == B, unpaired unequal
scalar t =r(t)
sum A, meanonly
replace A=A-r(mean) + 1.110498 ///1.110498=combined mean of A and B
sum B, meanonly
replace B=B-r(mean) + 1.110498
bootstrap r(t), reps(10000) nodots///
saving(C:\Users\user\Desktop\Czarina.dta, every(1) double replace) : ///
ttest A == B, unpairedunequal
use "C:\Users\user\Desktop\Czarina.dta", clear
count if _bs_1<=-1.4857///-1.4857=t-value from parametric ttest
count if _bs_1>=1.4857
display (811+732)/10000///this chunk of code calculates a bootstrap p-value///
to be compared with the parametric ttest p-value
share|improve this answer

There are two different ways to justify the use of the t-test.

  • Your data is normally distributed and you have at least two samples per group
  • You have large sample sizes in each group

If either of these cases hold, then the t-test is considered a valid test. So if you are willing to make the assumption that your data is normally distributed (which many researchers who collect small samples are), then you have nothing to worry about.

However, someone might reasonably object that you are relying on this assumption to get your results, especially if your data is known to be skewed. Then the question of sample size required for valid inference is a very reasonable one.

As for how large a sample size is required, unfortunately there's no real solid answer for that; the more skewed your data, the bigger the sample size required to make the approximation reasonable. 15-20 per group is usually considered reasonable large, but as with most rules of thumb, there exist counter examples: for example, in lottery ticket returns (where 1 in, say, 10,000,000 observations is an EXTREME outlier), you would literally need somewhere around 100,000,000 observations before these tests would be appropriate.

share|improve this answer

I concur regarding the usefulness of a boostrapped t-test. I would also recommend, as a comparison, a look at the Bayesian method offered by Kruschke at In general, questions of "How many subjects?" can't be answered unless you have in hand an idea of what a significant effect size would be in terms of the problem being solved. That is, and for instance, if the test were a hypothetical study regarding the efficacy of a new drug, the effect size might be the minimum size needed to justify the new drug compared to old for the U.S. Food and Drug Administration.

What's odd in this and many other discussions is the wholesale willingness to posit that some data just have some theoretical distribution, like being Gaussian. First, we don't need to posit, we can check, even with small samples. Second, why posit any specific theoretical distribution at all? Why not just take the data as an empirical distribution unto itself?

Sure, in the case of small sample sizes, positing that the data come from some distribution is highly useful for analysis. But, to paraphrase Bradley Efron, in doing so you've just made up an infinite amount of data. Sometimes that can be okay if your problem is appropriate. Some times it isn't.

share|improve this answer

protected by Glen_b Jul 4 '15 at 18:29

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.