# What's the correct tool to compare means between very small groups?

I'm not really used to VERY small samples, so I'm wondering what to do in the following situation: I have 3 groups, A, B and C. Each of them consists of 3 observations. I want to test whether there is a difference, so actually a classical ANOVA problem, followed by some post-hoc analysis (I would like to use Tukey's test). However, I have only 3 observations per group, so no clue of normality. Moreover the variances are not more or less identical (actually they are quite different). So, should I switch to a non-parametric alternative? (Of course I know that the statistical analysis is not very meaningful with these sample sizes, the boxplots show it quite better, but a reviewer wanted to see some p-values as well and actually almost all my tests were significant...). Thanks!

edit: So first of all, just as an example, for the first comparison the variances are 1 vs 6 vs 36 in the 3 groups (the means are 8,32,35). So I don't have a real feeling whether they are VERY different. For some other comparisons they are closer, of course we have to interpret them related to the mean. So almost all of my ANOVAs are clearly significant (the differences in means are quite obvious), but I'm wondering whether I'm allowed to use these results. We don't want to do everything the reviewer wants, some things are just statistically non-sense here, but some analyses would be rather nice, and I like the results of the ANOVA and the TukeyHSD tests (here, e.g., A vs.B and A vs.C gets significant, as expected, but not B vs.C). It's completely clear when looking at the plots but sometimes it's nice to add a p-value...

I'd rather like using a basic, commonly used, approach (but it should be correct of course!) I'm a mathematician and not so experienced with real world applications yet,but what I learned is that for papers like these it's better to stick to the golden standard, if possible. Further I was wondering whether Kruskal-Wallis would make sense?

• Interesting question and a +1 from me, but you're allowed to tell a reviewer, "Thank you for the comment; that is inappropriate because [reasons]," and then not take their suggestion.
– Dave
Commented Mar 7, 2022 at 20:52
• Even with conditionally Normal responses and no group effect, the group variances can range widely. In your case, for instance, there is a 4.5% chance that the ratio of the largest to the smallest of those three variances exceeds 100. (This ratio could be further inflated when the common standard deviation is close to the precision with which values have been recorded.) So: just how different are the variances?
– whuber
Commented Mar 7, 2022 at 21:30
• That highlights the fundamental issue: either you perform your analysis believing in (approximate) Normality or you challenge that assumption. You can't challenge it through an examination or testing of the data, which are consistent with Normality (at least on the evidence of the group variances), leaving you to rely on other forms of evidence. Your options could range from doing what is ordinarily done in your scientific community, to applying your own beliefs, to examining behaviors of similar datasets, etc.
– whuber
Commented Mar 8, 2022 at 18:58
• @COOLSerdash That's a property of the ratio distribution of the range of the variances. It's useful to have some intuition for how this behaves because it is of practical help in evaluating ANOVA-like data. Here's an R simulation to check: n <- 1e1; ngroups <- 3; npergroup <- 3; X <- apply(array(rnorm(3*3*n), c(npergroup,ngroups,n)), 3, function(x) range(apply(x, 2, var))); r <- X[2,] / X[1,] I recommend examining a histogram of the log ratio log(r) or computing quantiles.
– whuber
Commented Mar 9, 2022 at 15:47
• @whuber You're right, I think the exact probability is 50/1139. Commented Mar 9, 2022 at 16:21

Even if there are real differences among group population means, there is no guarantee than an ANOVA with such modest numbers of replications, after post hoc comparisons, will allow you to conclude which of the three group means is largest and which is smallest.

If your variances seem obviously different, I suggest using oneway.test in R (or similar procedure), which does not assume equal variances in the three levels of the factor. Consider the following fictitious data, very roughly modeled after the description in the you Question:

set.seed(2022)
x1 = rnorm(3, 50, 4)
x2 = rnorm(3, 55, 6)
x3 = rnorm(3, 60, 5)
stripchart(list(x1,x2,x3), pch=19)


x = c(x1,x2,x3);  g = c(1,1,1,2,2,2,3,3,3)
oneway.test(x~g)

One-way analysis of means
(not assuming equal variances)

data:  x and g
F = 5.2598, num df = 2.0000, denom df = 3.8527, p-value = 0.07918


There are significant differences at the 8% level, but not at the 5% level, even though all observations in Group 3 are larger than any observation in Group 2. An ordinary ANOVA would have had 6 denominator degrees of freedom--reduced to about 4 here on account of differing group variances.

The following simulation shows that the power of oneway.test with such data is poor: only about 30% (more precisely $$0.302 \pm 0.003).$$ So, you will be doing ad hoc tests only occasionally.

set.seed(307)
pv = replicate(10^5,
oneway.test(c(rnorm(3,50,4),rnorm(3,55,6),rnorm(3,60,5))
~ g)\$p.value)
mean(pv <= 0.05)
[1] 0.30242       # aprx power
2*sd(pv <= 0.05)/sqrt(10^5)
[1] 0.002904922   # aprx margin of simulation error

• I tried oneway.test and the results are very similar- however, I'm more interested in individual comparisons, so which post-hoc analysis would be suitable here? Commented Mar 8, 2022 at 15:24
• Generally speaking, no post hoc analysis is valid in an ANOVA unless you have a significant main effect. With three replications per level and heteroscedasticicity I doubt you will legitimately see many significant main effects. (As shown in my simulation.) If so, be sure to use some method of avoiding false discovery (such as Tukey or Bonferroni). // You need more replications per level before you can routinely identify the best and worst of three levels. // Before more experimentation, you need to do a power and sample size analysis to improve chances of detecting real effects. Commented Mar 8, 2022 at 19:45
• You edited your original question with more information. I have some additional questions based on that: How can you make a useful boxplot with three observations? With such severe differences among variances, what do you think a Kruskal-Wallis test is telling you? The power of most rank-based nonparametric tests (including K-W) is poor with such small sample sizes. One simulation of K-W tests and the latest description of your data shown "power" less than 20%--for whatever K-W is doing. // I don't want to be entirely negative. Your intuition may be OK.@whuber has provided some suggestions. Commented Mar 8, 2022 at 20:49
• Sorry, I was wrong- Ididn't mean boxplots, I meant barplots. The bars just show the mean value of the three and there's an error bar representing the standard error of the mean. So you get an idea of how much variance there actually is in the data. I know that's hard to do some meaningful statistic with these sample sizes and I still favour the plots, not the tests. However, I get numerous significant p-values, possible because differences are in most of the cases really large. Commented Mar 9, 2022 at 20:35
• I was just curious to see what Levene test does and there was no signficance. I think, this results from the really small samples because the variances are indeed different. My knowledge about all these tests is rather theoretical. I don't have enough real-world-experience to decide which procedure is the best in this case. Commented Mar 9, 2022 at 20:35