In practice we sometimes used a wisdom that two overlapping CIs for sample mean distribution meant we could not reject Hull hypothesis, to get a quick answer.
It looks though that this is a very raw estimate. I was working on a toy example of a two-sample hypothesis testing when I realized that our "wisdom" was just incorrect.
Am I right?
When I took the difference of samples X and Y, and plotted its moments, apparently the p-value is quite low.
α = 0.01; one-sided test; n(X) = n(Y) = 1000; var(X) = var(Y) = 1.
> t.test(x = xy[sample == 'x', x], y = xy[sample == 'y', x],
+ alternative = "less",
+ mu = 0, paired = F, var.equal = T)
Two Sample t-test
data: xy[sample == "x", x] and xy[sample == "y", x]
t = -2.7023, df = 1998, p-value = 0.003472
alternative hypothesis: true difference in means is less than 0
95 percent confidence interval:
-Inf -0.04665049
sample estimates:
mean of x mean of y
0.02546606 0.14476431
