# Why do I have different confident level when using TukeyHSD, t.test and formula in R

I have a data, including two groups. I need to calculate the difference of these two groups, then calculate the confident level of 0.9. Also, I need to identify if these two groups are significantly different using ANOVA. However, I find something weird when calculating confident level. Here is my data

Sample <- data.frame(a=runif(100,0,1000),b=runif(100,0,1000))


The real data is too many, so I generate some random data.

First of all, I use the definition of confident level to calculate:

Diff <- (Sample$b-Sample$a)
Sd <- sd(Diff)
Mean <- mean(Diff)
Me <- qnorm(.9)*Sd/sqrt(length(Diff))
Lower <- Mean-Me
Upper <- Mean+Me


The result is:

> Lower
[1] -61.48252
> Upper
[1] 54.21919


Second of all, I use aov to do ANOVA analysis, then TukeyHSD is used to show the confident level:

Sample1 <- Sample %>%
gather(Group,Score)
Aov <- aov(Score~Group,Sample1)
TukeyHSD(Aov,conf.level = .9)
> TukeyHSD(Aov,conf.level = .9)
Tukey multiple comparisons of means
90% family-wise confidence level
Fit: aov(formula = Score ~ Group, data = Sample1)
\$Group
b-a -3.631661 -76.10958 68.84626 0.9340892


Third of all, I use t.test to calculate the confident level of the difference of two groups:

> t.test(Diff,conf.level = .9)
One Sample t-test
data:  Diff
t = -0.080451, df = 99, p-value = 0.936
alternative hypothesis: true mean is not equal to 0
90 percent confidence interval:
-78.58381  71.32049
sample estimates:
mean of x
-3.631661


Because of randomly generating data, you probably have the different sample. But anyway, the confident level from three methods are different. S, am I doing wrong? And what is the right way?

• your first confidence interval 'Me <- qnorm(.9)*Sd/sqrt(length(Diff)' is too narrow. When creating a two-sided confidence interval of alpha=0.9, you need to 'divide' the uncertainty onto both the lower and the upper side. So the .9 or 90% confidence interval would be 'Me <- qnorm(.95)*Sd/sqrt(length(Diff)' – IWS Oct 11 '17 at 7:12
• Could you also explain why these results are different? Thanks – Feng Chen Oct 11 '17 at 10:01

When you create the variable Diff, you are creating a single vector of numbers.

a = c(1,2,3,4,5,6,7,8,9,10)

b = c(2,4,6,8,10,12,14,16,18,20)

Diff = b - a

Diff

### [1]  1  2  3  4  5  6  7  8  9 10


The t.test function will return the confidence interval for this single vector.

t.test(Diff)

### 95 percent confidence interval:
### 3.334149 7.665851


This is the same as doing a paired t-test on the two groups, a and b.

t.test(b, a, paired=TRUE)

### 95 percent confidence interval:
### 3.334149 7.665851


To replicate this by hand, you need to 1) use the t-distribution, and 2) use 1/2 of alpha. 9 is the degrees of freedom for this example.

Sd    <- sd(Diff)
Mean  <- mean(Diff)
Me    <- qt(.025, 9, lower.tail=FALSE)*Sd/sqrt(length(Diff))
Lower <- Mean - Me
Upper <- Mean + Me

Lower

Upper

### Lower
### [1] 3.334149

### Upper
### [1] 7.665851


TukeyHSD is treating the Groups as two independent samples, not as a single vector of numbers, or as the difference in paired samples. So it is different conceptually to the previous calculations.