I have two questions about Bonferroni adjustments:

1). Can one use the Bonferroni method to compare independent groups? The reason why I ask this is it seems that many examples I've encountered discuss the Bonferroni method in the context of comparing dependent groups - for example, multiple comparisons after repeated measures ANOVA.

2). I created a set of simulated data (see code below for the reproducible dataset).

data<-data.frame(x=rep(letters[1:4], each=5), y=sort(rlnorm(20)))

Then, I used pairwise.t.test() and set p.adj="bonf" (see below) to test pairwise comparisons.

pairwise.t.test(x=data$y, g=data$x, p.adj="bonf") #see results below:

#  data:  data$y and data$x 
#    a       b       c      
#  b 1.00000 -       -      
#  c 0.38945 1.00000 -      
#  d 8.3e-06 3.5e-05 0.00031

# P value adjustment method: bonferroni 

However, these results are different from the results obtained by doing pairwise t-tests using t.test() and then adjusting for the p-values (see below)

t.test(y~x, data[data$x=="a" | data$x=="b",])$p.value*6
t.test(y~x, data[data$x=="a" | data$x=="c",])$p.value*6
t.test(y~x, data[data$x=="a" | data$x=="d",])$p.value*6
t.test(y~x, data[data$x=="b" | data$x=="c",])$p.value*6
t.test(y~x, data[data$x=="b" | data$x=="d",])$p.value*6
t.test(y~x, data[data$x=="c" | data$x=="d",])$p.value*6

The results are below:

# a vs. b = 0.0788128848           
# a vs. c = 0.0001770066           
# a vs. d = 0.0324680659
# b vs. c = 0.0137812904
# b vs. d = 0.0488036762
# c vs. d = 0.0970799045

These adjusted p-values are rather different from the ones obtained from individual t-tests. So I wonder why there are such big differences.

  • 2
    $\begingroup$ For your 2nd question, you must set pool.sd=FALSE in pairwise.t.test if you want to get comparable results; otherwise R is using a pooled estimate of variance based on your three treatments which obviously differs from the one used in t-test for 2 independent samples. $\endgroup$
    – chl
    Feb 27 '13 at 8:08
  • $\begingroup$ uh, thanks! I thought I used pool.sd=FALSE and it didn't give me the right results either. As it turned out, for the whole time, I spelled the argument incorrectly as pooled.sd, and I didn't get any error message... $\endgroup$
    – Alex
    Feb 27 '13 at 8:16
  • $\begingroup$ Regarding question one: There is no need to adjust standard errors for multiple testing if you have one (independent) group for each hypothesis you are testing. The issue arises if you are testing multiple hypothesis with one group. $\endgroup$ Feb 27 '13 at 8:29
  • $\begingroup$ I'm not R user, but I created your data here and checked in SPSS. Please - for the future - try (if possible) to give the data itself, not R code which not everybody can read or use. $\endgroup$
    – ttnphns
    Feb 27 '13 at 8:41
  1. Yes, one can use Bonferroni for independent groups.
  2. There is two issues. First issue. Your first results are correct ANOVA post-hoc results. The variance is estimated by pooling from all 4 groups and then this estimate is used in each of the pairwise tests. In your second results, you seem to estimate variance only from the two groups being currently compared. Second issue. In your second results, you're using not classic Student's t-test but its Welch's version which does not assume equal variances for the groups in the population.
  • $\begingroup$ (+1) Your point 2 is more accurate than my quick comment. $\endgroup$
    – chl
    Feb 27 '13 at 9:59
  • $\begingroup$ Is it possible to refer me to some article about pooled variance? It seems that everything I've read about Bonferroni correction simply says that you do a bunch of t-tests and simply adjust the alpha value or the p-value. Thanks! $\endgroup$
    – Alex
    Feb 27 '13 at 14:12
  • $\begingroup$ Also, if I actually have a set of a priori comparisons to make - say, instead of making 6 comparisons, I am only interested in 3 comparisons for theoretical reasons, will the pooled variance be different? $\endgroup$
    – Alex
    Feb 27 '13 at 14:20
  • 1
    $\begingroup$ @Alex, it's not about Bonferroni correction (you applied it correctly), it's about the t-test. You can use it either as an ANOVA post-hoc set of multiple comparisons or independently pairwisely. In the latter way, you may choose between several modifications of the test. $\endgroup$
    – ttnphns
    Feb 27 '13 at 14:31
  • 2
    $\begingroup$ When you compare any two groups you may prefer either way: use variances only of those two groups or use the pooled variance of all the groups or subset of the groups. The choice depends on your task and reasoning and is often not obvious. $\endgroup$
    – ttnphns
    Feb 27 '13 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.