two-sample t-test VS two one-sample t-tests. What's the difference?

Question

I have dataset which contains values of some numeric variable for two independent groups: mean sd n group 1: 5.094294 0.9678155 17 group 2: 4.614275 0.9080703 230 So, I want to check hypothesis that the mean of group 1 is greater than the mean of group 2 with 95% confidence level.

1. As far as I know, the most common way do this is two sample t-test. I'd checked it for both var.eual = TRUE and FALSE. This approach leads to accepting of my hypothesis (mean of gr1 is greater than mean of gr2).

t.test(gr1, gr2,paired = FALSE, var.equal = TRUE, alternative = "greater",conf.level = 0.95)

Two Sample t-test
data: gr1 and gr2
t = 2.0939, df = 245, p-value = 0.01865
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
0.1015146 Inf
sample estimates:
mean of x mean of y
5.094294 4.614275

t.test(gr1, gr2,paired = FALSE, var.equal = FALSE, alternative = "greater",conf.level = 0.95)

Welch Two Sample t-test
data: gr1 and gr2
t = 1.9815, df = 18.145, p-value = 0.03145
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
0.06013003 Inf
sample estimates:
mean of x mean of y
5.094294 4.614275

2. On the other hand we can separately calculate 95% confidence interval for mean of each group. If this confidence intervals have intersection, then we'll be able to claim that means are not different (for 95% confidence level). This approach leads to rejection of my hypothesis.

t.test(gr1,alternative = "greater")

One Sample t-test
data: gr1
t = 21.7028, df = 16, p-value = 1.353e-13
alternative hypothesis: true mean is greater than 0
95 percent confidence interval:
4.684483 Inf
sample estimates:
mean of x
5.094294

t.test(gr2,alternative = "less")

One Sample t-test
data: gr2
t = 77.0634, df = 229, p-value = 1
alternative hypothesis: true mean is less than 0
95 percent confidence interval:
-Inf 4.713163
sample estimates:
mean of x
4.614275

Can somebody explain me why this two approaches give different results? Which of approaches is better to use and why?

Answer 1

The two-sample t-test is appropriate here, because you want to compare the two groups directly.

Two groups can differ significantly, and yet the CIs can still overlap. However, if the CIs do not overlap, then the groups must differ significantly. (This is of course assuming that the significance test and the CIs are calculated using the same assumptions about the data.) This is commonly misunderstood. Reference: http://blog.minitab.com/blog/real-world-quality-improvement/common-statistical-mistakes-you-should-avoid

How can the means of two groups differ significantly and yet have overlapping CIs? Loosely speaking, I think of it this way. There is 95% likelihood that the true mean for each group lies within the CI for that group. But in order for them to have the same mean, one group mean would lie at the extreme of its CI, and the other group mean would lie at the opposite extreme of its CI. That is an unlikely scenario.

Answer 2

Here is some example data where the confidence intervals of the means overlap, but the difference between them is still significant using the two-sample t-test. The dotted lines are the 95% confidence intervals about the means, the solid lines are the bootstrap CI estimates.

library(ggplot2)
library(reshape2)

## Generate some sample data
set.seed(100)
dat <- data.frame(gr1=rnorm(100, 13, 20), gr2=rnorm(100, 8, 15))
p <- with( dat, t.test(gr1, gr2, alternative = "greater") )$p.value              # two-sample p.value
d <- melt(dat)

ggplot(d, aes(x=variable, y=value)) + 
  geom_point(alpha=0.5, position=position_jitter(width=0.1)) +
  stat_summary(fun.data="mean_cl_boot", col=c("blue", "red")) +                  # bootstrapped CIs
  geom_hline(y=sd(dat$gr1)/10*c(-1,1)*1.96 + mean(dat$gr1), col="blue", lty=2) + # CI for gr1 (95%)
  geom_hline(y=sd(dat$gr2)/10*c(-1,1)*1.96 + mean(dat$gr2), col="red", lty=2) +  # CI for gr2
  annotate(geom="text", x=2, y=45, label=paste("Two-sample p=", round(p,3)))

With the one-sample tests you are just asking if the mean is significantly different (or less/greater) from some value (defaults to 0), which in these cases it obviously is. You could do a one-sample t-test to see if the mean of group 1 is greater than the mean of group 2 by

t.test(dat$gr1, mu=mean(dat$gr2), alternative="greater")

but this wouldn't be correct, because you would be treating the mean of group 2 as a fixed value (no variation). So, the appropriate test would be the two-sample t-test, in which you account for variation in both populations. In the two-sample you are testing to see if the difference of the means is significantly different than 0, or since you selected greater, to see if difference is significantly greater than 0. With the Looking at this data, that might not be apparent, since the means are quite close and the populations are variable.

The 95% confidence interval about the mean is simply saying, "if we run this experiment 100 times, we expect this interval to contain the population mean 95 times." There is no probability or likelihood of the population mean taking on different values within the interval, because it is not a random variable, it is assumed to be fixed.