8
$\begingroup$

Following the design and data described in this question, I did a simple one-way within-subjects repeated-measures (RM) ANOVA and found some significant p-values. I then applied non-orthogonal post-hoc Tukey's HSD tests, and when I got significant results I applied Holm-Bonferroni (1979) correction. Whenever some p-values survived the FWER correction, I calculated 95% CIs and mean for the associated pairwise comparisons.

My question is: If I don't observe a significant result at any of the above steps, do I have to carry out a power analysis for the RM ANOVA, apply Tukey's HSD test or Holm-Bonferroni adjustments, or do I simply report results from the RM ANOVA without doing the power analysis?

The problem is that I'm starting to immerse in biostatistics only after my experiments, and unfortunately I didn't run a power analysis beforehand.

Improve this question
$\endgroup$
1
  • $\begingroup$ @chl thanks for editing :). but in the part "I calculated 95% CIs and mean for the associated pairwise comparisons" I did 95% CI and mean for each group compared. But I remembered that the difference between means of dependent groups compared and the 95% CI for the difference is appropriate. So you're right! But how to calculate it in R? $\endgroup$
    – abc
    Commented Jun 20, 2011 at 11:41

4 Answers 4

Reset to default
17
$\begingroup$

The hardline view on post-hoc power calculation is: don't do it as it's pointless. Russ Lenth from the University of Iowa has an article on this topic here (He also has an amusingly facetious Java applet for post-hoc power on his website).

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ Further discussion and references can also be found on p.2 in the article describing G*Power3: psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/… (pdf) $\endgroup$
    – caracal
    Commented Jun 20, 2011 at 13:48
  • $\begingroup$ @Freua: We do post hocs for check for a significance after an ANOVA positive answer, do we? I also considered an article by N.Colegrave & G.D.Ruxton. As a beginner I can say there are many "buts" :). Eventually we can speculate about (non)parametrics $\endgroup$
    – abc
    Commented Jun 20, 2011 at 21:27
8
$\begingroup$

As an aside, Tukey's doesn't depend on the ANOVA results being significant; you can have significant pairwise differences even when the overall ANOVA is not significant.

That is to say, if you're going to be doing Tukey-corrected pairwise comparisons, don't bother checking for overall significance first. If you only run the Tukey comparisons after getting a significant overall p-value, you are over-correcting.

(I'm confident that this is true with regular ANOVA; it's possible that with repeated measures or non-orthogonality something else happens; anyone care to chime in?)

Finally, to agree with Freya but to provide a little more guidance, instead of a post-hoc power test, a more reasonable thing to report would be the confidence intervals; they show exactly how big a difference your experiment could have detected, which is usually what people are after when they want a post-hoc power test anyway.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer. This is something that interests me, because some text books say that the Tukey HSD comparisons should not be done if the overall test is not significant. I've done a few simulations which showed that the Tukey HSD confidence intervals have the correct coverage regardless of significance of the overall test, so I don't quite understand the recommendation to only use them if the overall test is significant. I'll be interested to hear some other opinions. $\endgroup$
    – mark999
    Commented Jun 21, 2011 at 7:07
  • $\begingroup$ definitely with you on reporting CIs. The article by Colegrave & Ruxton that @stan cited has a nice explanation of this, which I'll quote: "What we are interested in is the description of the possible effect sizes that are supported by the data that we have, and the possible effect sizes that are not supported. [...] If the test was nonsignificant, then the confidence interval for effect size will span zero. However the breadth of that confidence interval gives an indication of the likelihood of the real effect size being zero." $\endgroup$
    – Freya Harrison
    Commented Jun 21, 2011 at 9:14
3
$\begingroup$

Another good discussion of the pitfalls of post-hoc power estimation is found in:

Gerard, P. D., D. R. Smith, and G. Weerakkody. 1998. Limits of retrospective power analysis. Journal of Wildlife Management 62:801-807 [link].

Improve this answer
$\endgroup$
0
2
$\begingroup$

Most text books argue that it is only proper to do a post hoc such as Tukey's only with a significant f. If you chose planned comparison based on theory, a non significant F would be okay ... Tukey's is a fairly conservative test that typically won't show significance if f is not significant. What value are you using for mean square within to calculate Tukey's? The confidence intervals are also supposed to use mean square with rather than separate variance estimates.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks for your reply. Excuse me, I'm not sure I understood the question... I use SAS with default settings including for mean square... Are there any advances beyond Tukye's HSD? $\endgroup$
    – abc
    Commented Aug 18, 2012 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.