2
$\begingroup$

In R, I was running pairwise t tests after a significant ANOVA. To understand better how pairwise t tests work, I set the method to adjust p-values to 'none', I was thus expecting to have exactly the same as if I was comparing 'manually' each group. However, this does not seem to be the case.

Why is that?

Here is an example.

My data:

df <- structure(list(treatment = structure(c(1L, 1L, 1L, 1L, 3L, 3L, 3L, 3L, 2L, 2L, 2L, 2L, 4L, 4L, 4L, 4L, 8L, 8L, 8L, 8L, 10L, 10L, 10L, 10L, 9L, 9L, 9L, 9L, 5L, 5L, 5L, 5L, 7L, 7L, 7L, 7L, 6L, 6L, 6L, 6L), .Label = c("oa", "oa_bhb_1000", "oa_bhb_500", "oa_bhb_5000", "oa_d_phe_10", "oa_d_phe_100", "oa_d_phe_50", "oa_l_phe_10", "oa_l_phe_100", "oa_l_phe_50"), class = "factor"), intensity = c(952343, 963296, 981994, 983969, 1258960, 918174, 1273620, 1281570, 1103510, 1154570, 1191380, 1063730, 974519, 948350, 911960, 892873, 1171440, 1066490, 1027280, 1004430, 1047480, 1185930, 987433, 1179250, 962765, 1107980, 1225000, 1054180, 1137860, 1239670, 998971, 1095830, 1166230, 1183520, 991854, 897242, 839749, 864329, 929698, 678285)), .Names = c("treatment", "intensity"), class = "data.frame", row.names = 41:80)

Running the ANOVA:

summary(aov (intensity ~ treatment, data = df))

p-value = 0.00102

Running the pairwise t tests without p-value adjustment:

pairwise.t.test (df$intensity, df$treatment, p.adjust.method = 'none')

Example, the p-value of the t test comparing treatment oa with treatment oa_bhb_500 is 0.00606.

But if I run this test manually:

t.test (df$intensity[1:4], df$intensity[5:8])

I get a p-value of 0.09493.

How can that be? Am I doing something wrong?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

Because the first one fits a model to the data for all the treatments and thus uses all the data to estimate the error variance; whereas the manually done test uses the data from only those two treatments.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I see. Should the second (the manual test) be less trusted you think? $\endgroup$
    – francoiskroll
    Commented Mar 31, 2018 at 9:39
  • 1
    $\begingroup$ If the assumption of equal population variances is true, then the pooled test is better because it uses all available information about that common variance. If the assumption is strongly violated, then the manual t tests are better. $\endgroup$
    – Russ Lenth
    Commented Mar 31, 2018 at 12:28
1
$\begingroup$

You need to disable pooled sd in the pairwise function to get the same result.

pairwise.t.test (df$intensity, df$treatment, p.adjust.method = 'none', pool.sd = FALSE)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.