When testing for a difference in mean between two conditions, biologists typically use a $t$-test, and wring their hands endlessly about how to justify removing outliers. Whereas I typically use a Mann-Whitney U test, which is robust to the presence of outliers.

The reason biologists can't just use a Mann-Whitney U test is that biological experiments typically have large effect sizes but extremely small samples. Often, only three measurements are made from each condition, for a total of 6 measurements in a two-sample test. This can easily be significant as long as the effect size is large, but the Mann-Whitney U test completely ignores the effect size. The U test cannot reach significance at the 0.05 level with only three measurements in each sample, no matter how large the difference is.

I think there has to be some test which is in some way "intermediate" between the $t$-test and a rank-based test, the way Huber estimators are intermediate between the mean and median. There has to be some way to do a test which is robust to outliers but also is able to gain power from a large effect size. Does anyone know of such a test?

To clarify: as stated in the title, I am looking for something I can use for comparing triplicate measurements of two conditions.

  • $\begingroup$ If you had 6 measurements and "outliers", I don't know how you would be able to conclude anything. (Or how you would even know what counts as an outlying value !) $\endgroup$ Dec 1, 2019 at 22:49
  • $\begingroup$ You might be on to something with the idea of Huber estimators. There are tests discussed by Rand Wilcox --- maybe in Wilcox RR (2017). Introduction to Robust Estimation & Hypothesis Testing. Elsevier. --- that use M-estimators. See also the R package WRS2. I don't know how well these approaches work with very small sample sizes. $\endgroup$ Dec 1, 2019 at 22:59
  • $\begingroup$ "Often, only three measurements are made from each condition, for a total of 6 measurements in a two-sample test." That itself is a problem. $\endgroup$
    – Galen
    Jun 17, 2021 at 1:42
  • $\begingroup$ @Galen the issue I see is that a medical experiment could be expensive or risk serious side effects like death, so minimizing the sample size is important. If we only care to detect large effects, perhaps $3\times 2$ observations is enough to give adequate power to a t-test. However, the Wilcoxon test cannot ever achieve $p\le 0.05$ with that sample size. $\endgroup$
    – Dave
    Jun 17, 2021 at 1:45
  • $\begingroup$ @Dave Yes, I am not saying there are not other factors to trade off. There's almost always a Pareto front of choices. I'm saying that really small sample sizes are themselves a problem. $\endgroup$
    – Galen
    Jun 17, 2021 at 1:49

1 Answer 1


Maybe you're looking for a permutation test. Here is a minimalist demonstration to get you started. (If interested, you may want to read more about permutation tests on this site and elsewhere.)

Data. Suppose you have four observations for each of Conditions 1 and 2, as follows:

x1 = c(100, 103, 110, 150)
x2 = c(140, 200, 205, 207)

x = c(x1, x2);  g = c(1,1,1,1, 2,2,2,2)
stripchart(x~g, ylim=c(.5, 2.5), pch=19)

enter image description here

Condition 1 tends to give smaller values, but both sets of data have what might be considered to be outliers, and one feels 'squishy' assuming normality to do a Welch two-sample t test (which does not require equal population variances).

Welch t test. The Welch test finds a significant difference with P-value $0.0127 < 0.05,$ as in the R output below, but we don't know whether to trust the result because assumptions might not be met.

t.test(x ~ g)

        Welch Two Sample t-test

data:  x by g
t = -3.645, df = 5.461, p-value = 0.0127
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -121.93616  -22.56384
sample estimates:
mean in group 1 mean in group 2 
         115.75          188.00 

Two-sample Wilcoxon test. A two-sample Wilcoxon test would be significant if all Condition 1 values were below any Condition 2 values, but not with our data, for which the P-value is $0.05714 > 0.05.$

wilcox.test(x ~ g)

        Wilcoxon rank sum test

data:  x by g
W = 1, p-value = 0.05714
alternative hypothesis: true location shift is not equal to 0

Permutation test. The Welch statistic $T$ (same as the pooled t statistic because the two sample sizes are equal) may be a reasonable quantitative way to express the difference between Condition 1 and 2 scores, even though the distribution of $T$ is in doubt.

If the null hypothesis is true, so that Conditions 1 and 2 tend to give the same results, it should not matter if we assign four of the eight observed values to Condition 1 at random, and the remaining four to Condition 2. We could find the Welch $T$ statistic for each of the ${8 \choose 4} = 70$ possible permuted assignments.

Then by brute force (perhaps aided a bit by combinatorics) we could find the value of $T$ for each of 70 possibilities, and thus the 'permutation distribution' of $T.$ Then we could decide whether the observed value of $T$ for the proper arrangement of observed values is sufficiently 'remarkable' to warrant rejection of the null hypothesis that the two Conditions are equivalent.

In practice, there might be many more than 70 possible arrangements and a complete combinatorial solution to the permutation distribution might be difficult to find. However, we can make many random permutations, find $T$ for each and thus use simulation to approximate the permutation distribution.

For our data the simulated permutation test can be done in R as shown below. For the seed (of the pseudorandom number generator) shown the P-value is approximately $0.03 < 0.05,$ so we can reject the null hypothesis. [Additional simulations with different seeds gave values 0.0282, 0.0300, 0.0279.]

t.obs = t.test(x ~ g)$stat
t = replicate(10^4, t.test(x ~ sample(g))$stat)
mean(abs(t) > abs(t.obs))
[1] 0.0298

Here is a histogram of the simulated permutation distribution of $T,$ with $\pm T_{obs}$ shown at vertical broken lines. The P-value is the proportion of simulated values of $T$ outside these lines.

hist(t, prob=T, col="skyblue2")
abline(v=c(t.obs,-t.obs), col="red", lwd=2, lty="dashed")

enter image description here

Indeed, the permutation distribution of $T$ does not look much like a t distribution, so our misgivings about using the Welch P-value are well-founded. But results of the permutation test clearly indicate that the null hypothesis should be rejected.

Notes: (1) A two-sample Wilcoxon test can be viewed as a 'frozen' permutation test. In part, the flexibility of the a general permutation test comes from the ability to choose different 'metrics' for expressing the difference Conditions (Welch t statistic, pooled t statistic, difference in sample means, difference in sample trimmed means, etc.)

(2) If you really have only three observations under each Condition, a permutation test may not be better choice than a two-sample Wilcoxon test for testing at the 5% level because there can be at most ${6 \choose 3} = 20$ distinct values in the permutation distribution.

  • 1
    $\begingroup$ Yeah, unless I'm missing something, any permutation test has the same problem as the Wilcoxon test. For triplicate measurements of two conditions, there's six observations and 20 permutations. Suppose we observe extremely strong evidence for A>B. One of the permutations is just exchanging the conditions, providing equally strong evidence A<B. For a two-sided test, the rejection region has to include both of these outcomes. That means the p value can never be lower than $2/\binom{6}{3} = 0.1$. $\endgroup$
    – user54038
    May 23, 2019 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.