Is it valid to conduct t-tests with just 20 students? [duplicate]

Question

By reading article I think I found a bug:

Seams to me that 20 is a very small number of samples in this article assuming 20 students are taken random and the order of students is not important. I read and it looks to me this question is super important but not treated with great care since it can be a reference to basically any sampling question.

Answer 1

Maybe it will help to look at a couple of examples with $n = 20$ differences. A paired t test on paired observations $(X_{1i}, X_{2i}), i=1,2,\dots,20,$ is the same as a one-sample t test on differences $Y_i = X_{2i} - X_{1i},$ measuring 'improvement' for the $i$th student.

Suppose $Y_i$ are as sampled in R below:

set.seed(225)
y = round(rnorm(20, 5, 5), 2)
summary(y);  length(y);  sd(Y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -3.060   1.948   4.925   5.208   8.000  14.620 
[1] 20         # sample size
[1] 0.5775886  # sample SD

A stripchart shows that the mean $\bar Y = 5.2$ is not only larger than $0$ but perhaps convincingly larger in terms of the variability of the 20 differences.

stripchart(y, pch="|")
 abline(v=0, col="green2", lty="dotted")

A t test of $H_0: \mu_d = 0$ against $H_a: \mu_d \ne 0$ shows a P-value very nearly $0$ so that we reject $H_0$ in favor of $H_a$ at the 5% level of significance (and at some smaller levels of significance).

t.test(y)

        One Sample t-test

data:  y
t = 5.2565, df = 19, p-value = 4.497e-05
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 3.134586 7.282414
sample estimates:
mean of x 
   5.2085 

In this one instance, a t test based on $n = 20$ observations from a normal population with $\mu$ one standard deviation $\sigma$ above $0$ seems to have plenty of power to detect that $\mu > 0.$

The particular sample above was not extraordinary in this regard, as the brief simulation below in R shows. About 99% of t tests on such data will reject $H_0;$ the power of the t test in these circumstances is about 99%.

set.seed(2021)
pv = replicate(10^5, t.test(rnorm(20,5,5))$p.val)
mean(pv <= 0.05)
[1] 0.98871

Here are results from a formal 'power and sample size' procedure in Minitab. (There is an R library that does such computations for a variety of commonly used tests, many kinds of statistical software have similar procedures, and there are web sites with reliable 'calculators'.) The accompanying power curve, shows useful power even for differences as small as $3.$

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus > null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 5

            Sample
Difference    Size     Power
         5      20  0.996103

However, it is hardly a surprise that the following sample does not happen to lead to rejection of $H_0.$

set.seed(226)
y = round(rnorm(20, 2, 5), 2)
mean(y);  sd(y)
[1] 1.2965
[1] 4.760695
t.test(y)$p.val
[1] 0.2381652   # fail to rej at 5% level

Speaking more technically, the power of a t test can be computed using a noncentral t distribution as discussed here.

An advantage of paired designs such as yours is that the variability of differences before and after tends to be small compared with the variability of the performances of individuals considered only before (or only after).

Addendum per comment:

set.seed(2021)
pv = replicate(10^5, t.test(rnorm(20,5,10), alt="g")$p.val)
mean(pv <= .05)
1] 0.69621

With $n = 20$ observed differences from $\mathsf{Norm}(\mu=5, \sigma=10),$ you will have power only about 70% of rejecting the null hypothesis (right-sided test).

Minitab output (without graph):

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus > null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 10

            Sample
Difference    Size     Power
         5      20  0.695149