In practice we sometimes used a wisdom that two overlapping CIs for sample mean distribution meant we could not reject Hull hypothesis, to get a quick answer.
It looks though that this is a very raw estimate. I was working on a toy example of a two-sample hypothesis testing when I realized that our "wisdom" was just incorrect.
Am I right?
When I took the difference of samples X and Y, and plotted its moments, apparently the p-value is quite low.
α = 0.01; one-sided test; n(X) = n(Y) = 1000; var(X) = var(Y) = 1.
> t.test(x = xy[sample == 'x', x], y = xy[sample == 'y', x], + alternative = "less", + mu = 0, paired = F, var.equal = T) Two Sample t-test data: xy[sample == "x", x] and xy[sample == "y", x] t = -2.7023, df = 1998, p-value = 0.003472 alternative hypothesis: true difference in means is less than 0 95 percent confidence interval: -Inf -0.04665049 sample estimates: mean of x mean of y 0.02546606 0.14476431