6
$\begingroup$

Following the design and data described in this question, I did a simple one-way within-subjects repeated-measures (RM) ANOVA and found some significant p-values. I then applied non-orthogonal post-hoc Tukey's HSD tests, and when I got significant results I applied Holm-Bonferroni (1979) correction. Whenever some p-values survived the FWER correction, I calculated 95% CIs and mean for the associated pairwise comparisons.

My question is: If I don't observe a significant result at any of the above steps, do I have to carry out a power analysis for the RM ANOVA, apply Tukey's HSD test or Holm-Bonferroni adjustments, or do I simply report results from the RM ANOVA without doing the power analysis?

The problem is that I'm starting to immerse in biostatistics only after my experiments, and unfortunately I didn't run a power analysis beforehand.

$\endgroup$
  • $\begingroup$ I tried to reword your question and provide a more illustrative title. Please, check that I didn't alter its original meaning. $\endgroup$ – chl Jun 20 '11 at 10:33
  • $\begingroup$ @chl thanks for editing :). but in the part "I calculated 95% CIs and mean for the associated pairwise comparisons" I did 95% CI and mean for each group compared. But I remembered that the difference between means of dependent groups compared and the 95% CI for the difference is appropriate. So you're right! But how to calculate it in R? $\endgroup$ – stan Jun 20 '11 at 11:41
  • $\begingroup$ @chl I changed the title... Probably it is OK... $\endgroup$ – stan Jun 20 '11 at 11:48
14
$\begingroup$

The hardline view on post-hoc power calculation is: don't do it as it's pointless. Russ Lenth from the University of Iowa has an article on this topic here (He also has an amusingly facetious Java applet for post-hoc power on his website).

$\endgroup$
  • 3
    $\begingroup$ Further discussion and references can also be found on p.2 in the article describing G*Power3: psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/… (pdf) $\endgroup$ – caracal Jun 20 '11 at 13:48
  • $\begingroup$ @Freua: We do post hocs for check for a significance after an ANOVA positive answer, do we? I also considered an article by N.Colegrave & G.D.Ruxton. As a beginner I can say there are many "buts" :). Eventually we can speculate about (non)parametrics $\endgroup$ – stan Jun 20 '11 at 21:27
7
$\begingroup$

As an aside, Tukey's doesn't depend on the ANOVA results being significant; you can have significant pairwise differences even when the overall ANOVA is not significant.

That is to say, if you're going to be doing Tukey-corrected pairwise comparisons, don't bother checking for overall significance first. If you only run the Tukey comparisons after getting a significant overall p-value, you are over-correcting.

(I'm confident that this is true with regular ANOVA; it's possible that with repeated measures or non-orthogonality something else happens; anyone care to chime in?)

Finally, to agree with Freya but to provide a little more guidance, instead of a post-hoc power test, a more reasonable thing to report would be the confidence intervals; they show exactly how big a difference your experiment could have detected, which is usually what people are after when they want a post-hoc power test anyway.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your answer. This is something that interests me, because some text books say that the Tukey HSD comparisons should not be done if the overall test is not significant. I've done a few simulations which showed that the Tukey HSD confidence intervals have the correct coverage regardless of significance of the overall test, so I don't quite understand the recommendation to only use them if the overall test is significant. I'll be interested to hear some other opinions. $\endgroup$ – mark999 Jun 21 '11 at 7:07
  • $\begingroup$ definitely with you on reporting CIs. The article by Colegrave & Ruxton that @stan cited has a nice explanation of this, which I'll quote: "What we are interested in is the description of the possible effect sizes that are supported by the data that we have, and the possible effect sizes that are not supported. [...] If the test was nonsignificant, then the confidence interval for effect size will span zero. However the breadth of that confidence interval gives an indication of the likelihood of the real effect size being zero." $\endgroup$ – Freya Harrison Jun 21 '11 at 9:14
1
$\begingroup$

Most text books argue that it is only proper to do a post hoc such as Tukey's only with a significant f. If you chose planned comparison based on theory, a non significant F would be okay ... Tukey's is a fairly conservative test that typically won't show significance if f is not significant. What value are you using for mean square within to calculate Tukey's? The confidence intervals are also supposed to use mean square with rather than separate variance estimates.

$\endgroup$
  • $\begingroup$ thanks for your reply. Excuse me, I'm not sure I understood the question... I use SAS with default settings including for mean square... Are there any advances beyond Tukye's HSD? $\endgroup$ – stan Aug 18 '12 at 22:25
1
$\begingroup$

Another good discussion of the pitfalls of post-hoc power estimation is found in:

Gerard, P. D., D. R. Smith, and G. Weerakkody. 1998. Limits of retrospective power analysis. Journal of Wildlife Management 62:801-807 [link].

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.