One difficulty (out of several) with far outliers in what
ought to be normal data is that the null hypothesis (no differences)
may be rejected too often, leading to false discovery.
For simplicity, I will illustrate with pooled 2-sample t tests--instead of ANOVAs.
Consider the following fictitious data comparing two samples of size 20
from the same exponential distribution. There is no difference between
the two populations, so a no test should reject the null hypothesis. We look first at a pooled 2-sample t test.
set.seed(2021)
x1 = rexp(20, 1)
x2 = rexp(20, 1)
x = c(x1,x2)
g = rep(1:2, each=20)
boxplot(x~g, horizontal=T)
Nevertheless, the pooled 2-sample t test is (narrowly) significant at the 5% level.
t.test(x~g, var.eq=T)
Two Sample t-test
data: x by g
t = -2.0987, df = 38, p-value = 0.04254
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-1.02119881 -0.01840687
sample estimates:
mean in group 1 mean in group 2
0.4915494 1.0113522
A simulation shows that this kind of false rejection occurs about
half the time two such exponential samples are compared with a pooled 2-sample t test.
pv = replicate(10^5, t.test(c(rexp(20,1),rexp(20,1))~g,
var.eq=T)$p.val)
mean(pv <= 0.5)
[1] 0.51002
By contrast, if we take ranks of the combined data, the ranks
will run from 1 to 40, so outliers are not likely. Yet, the relative
standing of the values in the two samples is preserved.
Consequently, the pooled 2-sample t test (correctly) does not reject.
boxplot(rank(x)~g, horizontal=T)
t.test(rank(x)~g, var.eq=T)
Two Sample t-test
data: rank(x) by g
t = -1.5413, df = 38, p-value = 0.1315
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-12.955283 1.755283
sample estimates:
mean in group 1 mean in group 2
17.7 23.3
For ranked data, a simulation shows that the pooled 2-sample t test
seldom rejects.
pv.r = replicate(105, t.test(rank(c(rexp(20,1),rexp(20,1)))~g,
var.eq=T)$p.val)
mean(pv.r <= .05)
[1] 0.01904762
In this case, the true rejection rate when there is no difference between the two populations is about 2%. Granted, about 5% would be better, but doing the pooled 2-sample t test on ranked data
is better than ignoring the skewness of exponential data and resuting
outliers.
