Consider two fictitious samples of size $n = 10$
from normal distributions.
set.seed(2021)
x1 = rnorm(10, 50, 7)
summary(x1); sd(x1)
Min. 1st Qu. Median Mean 3rd Qu. Max.
36.54 50.53 52.48 52.12 55.68 62.11
[1] 6.621324 # sample SD
x2 = rnorm(10, 60, 8)
summary(x2); sd(x2)
Min. 1st Qu. Median Mean 3rd Qu. Max.
45.27 58.63 66.65 63.88 72.10 72.99
[1] 10.08623
From stripcharts of individual values of the two samples
it would be difficult to judge whether the data are normal--if we did not know they were sampled from normal populations.
stripchart(list(x1,x2), ylim=c(.5,2.5), pch="|")
However, if you have reason to believe that the the data are
are normal (perhaps from past experience with similar data)
a two-sample t test seems a reasonable choice to test
$H_0: \mu_1 =\mu_2$ against $H_a: \mu_1 \ne \mu_2.$
Because I have no information in advance to suggest that
the two population variances are equal, I choose to do
a Welch two-sample t test, which does not require equal
variances.
For our particular datasets, the t test finds a significant
difference in means at the 1% level. For real data we would
never know absolutely for sure whether the population means differ. Here we happen to know that $\mu_1 = 50, \mu_2 = 60.$
t.test(x1, x2)
Welch Two Sample t-test
data: x1 and x2
t = -3.0806, df = 15.542, p-value = 0.007362
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-19.86148 -3.64602
sample estimates:
mean of x mean of y
52.12437 63.87811
To get a fuller picture whether the Welch two-sample t test
is appropriate to use on two such small normal samples,
we can look to see how many times we reject $H_0$ in 100,000
repetitions of such an experiment:
set.seed(404)
pv = replicate(10^5, t.test(rnorm(10,50,7),
rnorm(10,60,8))$p.val)
mean(pv <= 0.05)
[1] 0.79861
The answer is that the null hypothesis is rejected about 80% of the time. In technical language, we say that the power of the Welch t test in this situation, with sample sizes of ten, is about 80%. The
answer would be different if $\delta = |\mu_2 - \mu_1|$ were
smaller or larger than 10. Or if the population variances were
larger or smaller.
Notes: R code: The vector pv contains 100,000 P-values.
The logical vector pv <= 0.05 contains 100,000 TRUEa and FALSEs, and its mean is the proportion of its TRUEs.
Normality. If the stripchart of the samples looked like the one below, then I would be reluctant to do a t test because there seems to
be some evidence--even with samples as small as size ten--that the
data may not be normal. I have to admit this is a difficult
and subjective decision.
set.seed(123)
x1 = rexp(10, 1/50); x2 = rexp(10, 1/60)
stripchart(list(x1,x2), ylim=c(.5,2.5), pch="|")
It hardly helps that both of these samples fail a Shapio-Wilk normality test, which can be misleading for such small sample sizes.
shapiro.test(x1)$p.val
[1] 0.003781628
shapiro.test(x2)$p.val
[1] 0.0003721849
In practice, I guess one is more likely to be criticized for
having so little data than for showing results from t tests.
