Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

Do I have to use Bonferroni's or Holm's corrections? Please see this previous question for my example data and the design. (Shortly, I have three or more dependent groups.)

share|improve this question
@Andy W: thanks for you edit :) – stan Sep 10 '11 at 9:56

1 Answer

up vote 1 down vote accepted

As long as the meaning of alpha is the probability of a Type I error then any of your listed corrections can be applied to any of your tests. You might say that the corrections for Type II error are underlying test agnostic because they correct the alpha inflation, which would be constant across tests of different kinds.

In your case, where you've implied that you're starting with an ANOVA that shows there is something significant, then the correction can also depend on whether you had specific hypotheses about individual comparisons. I tend to like to just report the standard confidence intervals of the effects in question. I know this doesn't have any corrections for multiple comparisons and is subject to Type II error even with a significant ANOVA (it's equivalent to Fisher's PLSD) but I report it in such a way as to imply that future research should look "here" rather than saying my results are definitive on the matter.

It's hard to advise you on exactly what to do without seeing the pattern of results and also your idea of what should have happened (or whether it was entirely exploratory). Perhaps you want to craft a question that includes your analysis thus far, some of your hypotheses about what would happen, and maybe even a couple of reporting alternatives. Then you'll get some really good results on here. Better questions get better answers.

share|improve this answer
thank you for the answer :). Sorry, I can't understand it in terms of the question... :( Could you tell me more? OK, yes if 1way RM ANOVA doesn't tell me "There is a difference/We can't accept the H0", I stop here. Otherwise I proceed not with (planned) contrasts (as I didn't plan actually) but with (non-planned) post-hoc comparisons. This I do in case of normal distribution (or if I could transform my data into "normal"). But if I deal with nonnormal I do Friedman ANOVA and if I find anything significant I do the Wilcoxon test. But I have to control alpha... – stan Sep 10 '11 at 11:00
... So I do a correction. After ANOVA I do Tukey's HSD and then Holm. After Friedman I do the Wilcoxon's test. But what do I have to do after the latter? I was told that after Wilcoxon I had to do Bonferroni's, but I had done Holm's... Now I don't know what to do – stan Sep 10 '11 at 11:06
1  
You could do Bonferroni or Holm. Bonferroni is rather brutish. If you like Holm's logic then us that. And Tukey's HSD is a way of reporting that doesn't require an ANOVA first... thus it's probably conservative. When I have a significant RM ANOVA I tend to use standard confidence intervals in a specific way (see edited answer). – John Sep 10 '11 at 17:07
thank you for you replies you advised to "report the standard confidence intervals of the effects in question". Do you mean effect size ? (I didn't calculate the sample size for a selected effect size... If your answer is "yes" what should I do?) – stan Sep 26 '11 at 3:28
I've chosen Holm as the test is more powerful and I have dependent groups. But are there any more sensitive (powerful) tests appropriate for my situation? – stan Sep 26 '11 at 3:49
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.