I have a data, including two groups. I need to calculate the difference of these two groups, then calculate the confident level of 0.9. Also, I need to identify if these two groups are significantly different using ANOVA. However, I find something weird when calculating confident level. Here is my data

Sample <- data.frame(a=runif(100,0,1000),b=runif(100,0,1000))

The real data is too many, so I generate some random data.

First of all, I use the definition of confident level to calculate:

Diff <- (Sample$b-Sample$a)
Sd <- sd(Diff)
Mean <- mean(Diff)
Me <- qnorm(.9)*Sd/sqrt(length(Diff))
Lower <- Mean-Me
Upper <- Mean+Me

The result is:

> Lower
[1] -61.48252
> Upper
[1] 54.21919

Second of all, I use aov to do ANOVA analysis, then TukeyHSD is used to show the confident level:

Sample1 <- Sample %>%
Aov <- aov(Score~Group,Sample1)
TukeyHSD(Aov,conf.level = .9)
> TukeyHSD(Aov,conf.level = .9)
  Tukey multiple comparisons of means
    90% family-wise confidence level
Fit: aov(formula = Score ~ Group, data = Sample1)
        diff       lwr     upr    p adj
b-a -3.631661 -76.10958 68.84626 0.9340892

Third of all, I use t.test to calculate the confident level of the difference of two groups:

> t.test(Diff,conf.level = .9)
    One Sample t-test
data:  Diff
t = -0.080451, df = 99, p-value = 0.936
alternative hypothesis: true mean is not equal to 0
90 percent confidence interval:
 -78.58381  71.32049
sample estimates:
mean of x 

Because of randomly generating data, you probably have the different sample. But anyway, the confident level from three methods are different. S, am I doing wrong? And what is the right way?

  • $\begingroup$ your first confidence interval 'Me <- qnorm(.9)*Sd/sqrt(length(Diff)' is too narrow. When creating a two-sided confidence interval of alpha=0.9, you need to 'divide' the uncertainty onto both the lower and the upper side. So the .9 or 90% confidence interval would be 'Me <- qnorm(.95)*Sd/sqrt(length(Diff)' $\endgroup$ – IWS Oct 11 '17 at 7:12
  • $\begingroup$ Could you also explain why these results are different? Thanks $\endgroup$ – Feng Chen Oct 11 '17 at 10:01

When you create the variable Diff, you are creating a single vector of numbers.

a = c(1,2,3,4,5,6,7,8,9,10)

b = c(2,4,6,8,10,12,14,16,18,20)

Diff = b - a


### [1]  1  2  3  4  5  6  7  8  9 10

The t.test function will return the confidence interval for this single vector.


### 95 percent confidence interval:
### 3.334149 7.665851

This is the same as doing a paired t-test on the two groups, a and b.

t.test(b, a, paired=TRUE)

### 95 percent confidence interval:
### 3.334149 7.665851

To replicate this by hand, you need to 1) use the t-distribution, and 2) use 1/2 of alpha. 9 is the degrees of freedom for this example.

Sd    <- sd(Diff)
Mean  <- mean(Diff)
Me    <- qt(.025, 9, lower.tail=FALSE)*Sd/sqrt(length(Diff))
Lower <- Mean - Me
Upper <- Mean + Me



### Lower
### [1] 3.334149

### Upper
### [1] 7.665851

TukeyHSD is treating the Groups as two independent samples, not as a single vector of numbers, or as the difference in paired samples. So it is different conceptually to the previous calculations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.