# Is it valid to conduct t-tests with just 20 students? [duplicate]

By reading article I think I found a bug:

Seams to me that 20 is a very small number of samples in this article assuming 20 students are taken random and the order of students is not important. I read and it looks to me this question is super important but not treated with great care since it can be a reference to basically any sampling question.

• A $t$-test is designed to deal with small samples. A big issue with small samples is that tests are not powerful: they are more likely to fail to spot actual differences than tests with larger samples. In your described experiment, a greater concern might be that students could improve just by taking a second test without the guide and you test will not consider this Feb 25, 2021 at 12:27
• At this time I am just concerned with the number of samples, and your words "not powerful" since these are for reason. Just I would like to have some math quantification for this :) Feb 25, 2021 at 12:33
• The test is valid even for $n=2$ given the assumption of i.i.d. normal differences. Feb 25, 2021 at 12:50
• – whuber
Feb 25, 2021 at 19:03

Maybe it will help to look at a couple of examples with $$n = 20$$ differences. A paired t test on paired observations $$(X_{1i}, X_{2i}), i=1,2,\dots,20,$$ is the same as a one-sample t test on differences $$Y_i = X_{2i} - X_{1i},$$ measuring 'improvement' for the $$i$$th student.

Suppose $$Y_i$$ are as sampled in R below:

set.seed(225)
y = round(rnorm(20, 5, 5), 2)
summary(y);  length(y);  sd(Y)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
-3.060   1.948   4.925   5.208   8.000  14.620
[1] 20         # sample size
[1] 0.5775886  # sample SD


A stripchart shows that the mean $$\bar Y = 5.2$$ is not only larger than $$0$$ but perhaps convincingly larger in terms of the variability of the 20 differences.

stripchart(y, pch="|")
abline(v=0, col="green2", lty="dotted")


A t test of $$H_0: \mu_d = 0$$ against $$H_a: \mu_d \ne 0$$ shows a P-value very nearly $$0$$ so that we reject $$H_0$$ in favor of $$H_a$$ at the 5% level of significance (and at some smaller levels of significance).

t.test(y)

One Sample t-test

data:  y
t = 5.2565, df = 19, p-value = 4.497e-05
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
3.134586 7.282414
sample estimates:
mean of x
5.2085


In this one instance, a t test based on $$n = 20$$ observations from a normal population with $$\mu$$ one standard deviation $$\sigma$$ above $$0$$ seems to have plenty of power to detect that $$\mu > 0.$$

The particular sample above was not extraordinary in this regard, as the brief simulation below in R shows. About 99% of t tests on such data will reject $$H_0;$$ the power of the t test in these circumstances is about 99%.

set.seed(2021)
pv = replicate(10^5, t.test(rnorm(20,5,5))$p.val) mean(pv <= 0.05) [1] 0.98871  Here are results from a formal 'power and sample size' procedure in Minitab. (There is an R library that does such computations for a variety of commonly used tests, many kinds of statistical software have similar procedures, and there are web sites with reliable 'calculators'.) The accompanying power curve, shows useful power even for differences as small as $$3.$$ Power and Sample Size 1-Sample t Test Testing mean = null (versus > null) Calculating power for mean = null + difference α = 0.05 Assumed standard deviation = 5 Sample Difference Size Power 5 20 0.996103  However, it is hardly a surprise that the following sample does not happen to lead to rejection of $$H_0.$$ set.seed(226) y = round(rnorm(20, 2, 5), 2) mean(y); sd(y) [1] 1.2965 [1] 4.760695 t.test(y)$p.val
[1] 0.2381652   # fail to rej at 5% level


Speaking more technically, the power of a t test can be computed using a noncentral t distribution as discussed here.

An advantage of paired designs such as yours is that the variability of differences before and after tends to be small compared with the variability of the performances of individuals considered only before (or only after).

set.seed(2021)
pv = replicate(10^5, t.test(rnorm(20,5,10), alt="g")$p.val) mean(pv <= .05) 1] 0.69621  With $$n = 20$$ observed differences from $$\mathsf{Norm}(\mu=5, \sigma=10),$$ you will have power only about 70% of rejecting the null hypothesis (right-sided test). Minitab output (without graph): Power and Sample Size 1-Sample t Test Testing mean = null (versus > null) Calculating power for mean = null + difference α = 0.05 Assumed standard deviation = 10 Sample Difference Size Power 5 20 0.695149  • Great, I run all the code and I am trying now to sum all up: If you have pv = replicate(10^3, t.test(rnorm(20,5,10))$p.val) higher variance and relatively low number of samples like 20, like I altered your original example and set for instance $\sigma=10$, we cannot reject $H_0$ Feb 26, 2021 at 12:23
• Thanks @BruceET, if with stick with specific alternative parameter, it looks obvious to me that you can control the approval of rejection of the $H_0$ with exactly two numbers: number of samples $n$ in here it was 20 and standard deviation, 10 in the addendum, having significance level of 0.05 as the one that is well accommodated, I cannot recall who was the first person using significance level 0.05 it, but I guess last name started on either P. or N. Which can lead me to the spectacular quest: set.seed(2021) pv = replicate(10^5, t.test(rnorm(30,0,1))$p.val) mean(pv <= .05) Mar 3, 2021 at 19:33 • Of course things change with increasing amounts of data, but you are interpreting simulation results improperly. There is nothing wrong with using a t test for$n = 20$normal observations. The t test was introduced to be useful for small$n$. A t test would be OK for$n=3, 9, 15\$ etc. More data increases power of finding a difference if there is a real difference to be found. Mar 3, 2021 at 20:07