Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 7 down vote favorite
1

It strikes me that the available corrections for multiple comparisons in the context of a repeated measures ANOVA are excessively conservative. Is this actually the case? If so, what are some citations I can use to support this point and learn more?

share|improve this question
What kind of MP correction are you referring too? (BTW, why the alpha tag?) – chl♦ Jul 16 '11 at 20:51
re the alpha tag: Only because the corrections typically are meant to hold alpha constant. – Russell S. Pierce Jul 16 '11 at 23:04
I'm not sure what MP stands for in this context, but I'm talking about the frequently used Bonferroni and/or Sidak multiple comparisons corrections. – Russell S. Pierce Jul 16 '11 at 23:05
I know there are other approaches, e.g. False Discovery Rate and so forth, but I am specifically looking to evaluate and critique the more frequently used approaches. – Russell S. Pierce Jul 16 '11 at 23:07
Sorry, a typo of mine: By "MP correction", I meant MCP (multiple comparisons procedure). I had in mind Tukey HSD (which acts on the test statistic, not the p-value), but your question is interesting (my +1) because it call for a critical appraisal of recent literature on that problem. – chl♦ Jul 17 '11 at 11:06
show 2 more comments

2 Answers

up vote 4 down vote accepted

To the best of my knowledge, the joint distribution of linear contrasts has been derived in the simple ANOVA case (see the documentation of the multcomp R package), but there is are no closed forms for the repeated measures setup. Nevertheless, you can always Bootstrap the joint distribution of these linear contrasts under the null,and look at the minimal t-statistic (or maximal p-value) for setting the significance threshold with FWE control. As you also suggested, you can use methods which only require some qualitative condition on the joint distribution of the test statistics. Bonferroni is a good option if you have few contrasts. Otherwise, have a look at Holm's. If you are looking into many linear contrasts, you should definitely ask yourself you want protection from any false discovery or only a proportion of false discoveries. In the latter case, use the BH procedure for FDR control.

share|improve this answer
up vote 1 down vote

Here's a collection of links to a SPSS forum. Hope you find it relevant to you to some degree: this, this, this, this.

share|improve this answer
Thanks for pointing in a direction. Unfortunately news group posts are far from compelling evidence when discussing these issues in published work. The general primer on sphericity is interesting, and points out why Bonferroni-stype p-value adjustments are still in common use. Unfortunately, there isn't really an indictment of the conservative nature of that approach. – Russell S. Pierce Jul 17 '11 at 15:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.