By reading article I think I found a bug: enter image description here

Seams to me that 20 is a very small number of samples in this article assuming 20 students are taken random and the order of students is not important. I read and it looks to me this question is super important but not treated with great care since it can be a reference to basically any sampling question.

  • 4
    $\begingroup$ A $t$-test is designed to deal with small samples. A big issue with small samples is that tests are not powerful: they are more likely to fail to spot actual differences than tests with larger samples. In your described experiment, a greater concern might be that students could improve just by taking a second test without the guide and you test will not consider this $\endgroup$
    – Henry
    Feb 25, 2021 at 12:27
  • $\begingroup$ At this time I am just concerned with the number of samples, and your words "not powerful" since these are for reason. Just I would like to have some math quantification for this :) $\endgroup$ Feb 25, 2021 at 12:33
  • $\begingroup$ The test is valid even for $n=2$ given the assumption of i.i.d. normal differences. $\endgroup$ Feb 25, 2021 at 12:50
  • $\begingroup$ Our site search works pretty well for questions concerning basic statistical procedures. $\endgroup$
    – whuber
    Feb 25, 2021 at 19:03

1 Answer 1


Maybe it will help to look at a couple of examples with $n = 20$ differences. A paired t test on paired observations $(X_{1i}, X_{2i}), i=1,2,\dots,20,$ is the same as a one-sample t test on differences $Y_i = X_{2i} - X_{1i},$ measuring 'improvement' for the $i$th student.

Suppose $Y_i$ are as sampled in R below:

y = round(rnorm(20, 5, 5), 2)
summary(y);  length(y);  sd(Y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -3.060   1.948   4.925   5.208   8.000  14.620 
[1] 20         # sample size
[1] 0.5775886  # sample SD

A stripchart shows that the mean $\bar Y = 5.2$ is not only larger than $0$ but perhaps convincingly larger in terms of the variability of the 20 differences.

stripchart(y, pch="|")
 abline(v=0, col="green2", lty="dotted")

enter image description here

A t test of $H_0: \mu_d = 0$ against $H_a: \mu_d \ne 0$ shows a P-value very nearly $0$ so that we reject $H_0$ in favor of $H_a$ at the 5% level of significance (and at some smaller levels of significance).


        One Sample t-test

data:  y
t = 5.2565, df = 19, p-value = 4.497e-05
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 3.134586 7.282414
sample estimates:
mean of x 

In this one instance, a t test based on $n = 20$ observations from a normal population with $\mu$ one standard deviation $\sigma$ above $0$ seems to have plenty of power to detect that $\mu > 0.$

The particular sample above was not extraordinary in this regard, as the brief simulation below in R shows. About 99% of t tests on such data will reject $H_0;$ the power of the t test in these circumstances is about 99%.

pv = replicate(10^5, t.test(rnorm(20,5,5))$p.val)
mean(pv <= 0.05)
[1] 0.98871

Here are results from a formal 'power and sample size' procedure in Minitab. (There is an R library that does such computations for a variety of commonly used tests, many kinds of statistical software have similar procedures, and there are web sites with reliable 'calculators'.) The accompanying power curve, shows useful power even for differences as small as $3.$

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus > null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 5

Difference    Size     Power
         5      20  0.996103

enter image description here

However, it is hardly a surprise that the following sample does not happen to lead to rejection of $H_0.$

y = round(rnorm(20, 2, 5), 2)
mean(y);  sd(y)
[1] 1.2965
[1] 4.760695
[1] 0.2381652   # fail to rej at 5% level

Speaking more technically, the power of a t test can be computed using a noncentral t distribution as discussed here.

An advantage of paired designs such as yours is that the variability of differences before and after tends to be small compared with the variability of the performances of individuals considered only before (or only after).

Addendum per comment:

pv = replicate(10^5, t.test(rnorm(20,5,10), alt="g")$p.val)
mean(pv <= .05)
1] 0.69621

With $n = 20$ observed differences from $\mathsf{Norm}(\mu=5, \sigma=10),$ you will have power only about 70% of rejecting the null hypothesis (right-sided test).

Minitab output (without graph):

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus > null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 10

Difference    Size     Power
         5      20  0.695149
  • 1
    $\begingroup$ Great, I run all the code and I am trying now to sum all up: If you have pv = replicate(10^3, t.test(rnorm(20,5,10))$p.val) higher variance and relatively low number of samples like 20, like I altered your original example and set for instance $\sigma=10$, we cannot reject $H_0$ $\endgroup$ Feb 26, 2021 at 12:23
  • 1
    $\begingroup$ See addendum to my Answer. $\endgroup$
    – BruceET
    Mar 3, 2021 at 16:43
  • $\begingroup$ Thanks @BruceET, if with stick with specific alternative parameter, it looks obvious to me that you can control the approval of rejection of the $H_0$ with exactly two numbers: number of samples $n$ in here it was 20 and standard deviation, 10 in the addendum, having significance level of 0.05 as the one that is well accommodated, I cannot recall who was the first person using significance level 0.05 it, but I guess last name started on either P. or N. Which can lead me to the spectacular quest: set.seed(2021) pv = replicate(10^5, t.test(rnorm(30,0,1))$p.val) mean(pv <= .05) $\endgroup$ Mar 3, 2021 at 19:33
  • 1
    $\begingroup$ Of course things change with increasing amounts of data, but you are interpreting simulation results improperly. There is nothing wrong with using a t test for $n = 20$ normal observations. The t test was introduced to be useful for small $n$. A t test would be OK for $n=3, 9, 15$ etc. More data increases power of finding a difference if there is a real difference to be found. $\endgroup$
    – BruceET
    Mar 3, 2021 at 20:07
  • 1
    $\begingroup$ If you simulate data according to null hypothesis, then test at 5% level, then of course, you'll reject about 5% of the time. $\endgroup$
    – BruceET
    Mar 3, 2021 at 20:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.