Robust two-sample test with triplicate measurements?

Question

When testing for a difference in mean between two conditions, biologists typically use a $t$-test, and wring their hands endlessly about how to justify removing outliers. Whereas I typically use a Mann-Whitney U test, which is robust to the presence of outliers.

The reason biologists can't just use a Mann-Whitney U test is that biological experiments typically have large effect sizes but extremely small samples. Often, only three measurements are made from each condition, for a total of 6 measurements in a two-sample test. This can easily be significant as long as the effect size is large, but the Mann-Whitney U test completely ignores the effect size. The U test cannot reach significance at the 0.05 level with only three measurements in each sample, no matter how large the difference is.

I think there has to be some test which is in some way "intermediate" between the $t$-test and a rank-based test, the way Huber estimators are intermediate between the mean and median. There has to be some way to do a test which is robust to outliers but also is able to gain power from a large effect size. Does anyone know of such a test?

To clarify: as stated in the title, I am looking for something I can use for comparing triplicate measurements of two conditions.

Answer 1

Maybe you're looking for a permutation test. Here is a minimalist demonstration to get you started. (If interested, you may want to read more about permutation tests on this site and elsewhere.)

Data. Suppose you have four observations for each of Conditions 1 and 2, as follows:

x1 = c(100, 103, 110, 150)
x2 = c(140, 200, 205, 207)

x = c(x1, x2);  g = c(1,1,1,1, 2,2,2,2)
stripchart(x~g, ylim=c(.5, 2.5), pch=19)

Condition 1 tends to give smaller values, but both sets of data have what might be considered to be outliers, and one feels 'squishy' assuming normality to do a Welch two-sample t test (which does not require equal population variances).

Welch t test. The Welch test finds a significant difference with P-value $0.0127 < 0.05,$ as in the R output below, but we don't know whether to trust the result because assumptions might not be met.

t.test(x ~ g)

        Welch Two Sample t-test

data:  x by g
t = -3.645, df = 5.461, p-value = 0.0127
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -121.93616  -22.56384
sample estimates:
mean in group 1 mean in group 2 
         115.75          188.00 

Two-sample Wilcoxon test. A two-sample Wilcoxon test would be significant if all Condition 1 values were below any Condition 2 values, but not with our data, for which the P-value is $0.05714 > 0.05.$

wilcox.test(x ~ g)

        Wilcoxon rank sum test

data:  x by g
W = 1, p-value = 0.05714
alternative hypothesis: true location shift is not equal to 0

Permutation test. The Welch statistic $T$ (same as the pooled t statistic because the two sample sizes are equal) may be a reasonable quantitative way to express the difference between Condition 1 and 2 scores, even though the distribution of $T$ is in doubt.

If the null hypothesis is true, so that Conditions 1 and 2 tend to give the same results, it should not matter if we assign four of the eight observed values to Condition 1 at random, and the remaining four to Condition 2. We could find the Welch $T$ statistic for each of the ${8 \choose 4} = 70$ possible permuted assignments.

Then by brute force (perhaps aided a bit by combinatorics) we could find the value of $T$ for each of 70 possibilities, and thus the 'permutation distribution' of $T.$ Then we could decide whether the observed value of $T$ for the proper arrangement of observed values is sufficiently 'remarkable' to warrant rejection of the null hypothesis that the two Conditions are equivalent.

In practice, there might be many more than 70 possible arrangements and a complete combinatorial solution to the permutation distribution might be difficult to find. However, we can make many random permutations, find $T$ for each and thus use simulation to approximate the permutation distribution.

For our data the simulated permutation test can be done in R as shown below. For the seed (of the pseudorandom number generator) shown the P-value is approximately $0.03 < 0.05,$ so we can reject the null hypothesis. [Additional simulations with different seeds gave values 0.0282, 0.0300, 0.0279.]

set.seed(522)
t.obs = t.test(x ~ g)$stat
t = replicate(10^4, t.test(x ~ sample(g))$stat)
mean(abs(t) > abs(t.obs))
[1] 0.0298

Here is a histogram of the simulated permutation distribution of $T,$ with $\pm T_{obs}$ shown at vertical broken lines. The P-value is the proportion of simulated values of $T$ outside these lines.

hist(t, prob=T, col="skyblue2")
abline(v=c(t.obs,-t.obs), col="red", lwd=2, lty="dashed")

Indeed, the permutation distribution of $T$ does not look much like a t distribution, so our misgivings about using the Welch P-value are well-founded. But results of the permutation test clearly indicate that the null hypothesis should be rejected.

Notes: (1) A two-sample Wilcoxon test can be viewed as a 'frozen' permutation test. In part, the flexibility of the a general permutation test comes from the ability to choose different 'metrics' for expressing the difference Conditions (Welch t statistic, pooled t statistic, difference in sample means, difference in sample trimmed means, etc.)

(2) If you really have only three observations under each Condition, a permutation test may not be better choice than a two-sample Wilcoxon test for testing at the 5% level because there can be at most ${6 \choose 3} = 20$ distinct values in the permutation distribution.