2
$\begingroup$

I have an experiment with 3 equal group sizes and 4 measures. I think the simple null hypothesis is that the three groups will be the same. Most people, however, believe that group A should do best in measure 1, group B should do best in measure 2, group 3 should do best in measure C, and they should all be the same in measure 4, both when comparing within and between groups. It is quite possible my group size is too small to come to any conclusions. I base this on the tried and true approach that my N is smaller than their N.

I started off by doing a multivariate GLM (using SPSS) and found that the effect group is not significant (F = 1.8, p = 0.1). Looking at the data suggests that even if there is a difference between the groups, that it doesn't likely follow what is expected (e.g., group A does not do the best in measure 1 and is best in measure 2). My thinking is that maybe a post hoc power analysis might tell me something useful. SPSS spits out an observed power of 0.69. At this point I am lost. Does the observed power tell me anything useful? Am I going about this all wrong?

Improve this question
$\endgroup$
2
  • $\begingroup$ First of all, are these 4 measures understood to be independent, or are they measures of similar constructs? $\endgroup$
    – gung - Reinstate Monica
    Feb 11, 2013 at 22:12
  • $\begingroup$ @gung ummmm? They are measured on the same scale, but represent different things. They are neither highly correlated nor uncorrelated. $\endgroup$
    – StrongBad
    Feb 11, 2013 at 22:23

2 Answers 2

Reset to default
10
$\begingroup$

Post hoc power analyses are at best useless and are often misleading, read "The Abuse of Power" (Hoenig and Heisey, American statistician, vol 55, issue 1, 2001) for more details.

What might be more useful is confidence intervals on your measures, they can tell you if your original ideas could still be meaningful and if the plausible differences are large enough to care about. These will be much more meaningful than a transformation of the p-value (which is all that the post hoc power is) that is usually interpreted wrongly.

Improve this answer
$\endgroup$
1
$\begingroup$

SPSS observed power analysis for ANOVA with more than 2 groups provides misleading information about power. This can be easily seen by the fact that you obtained a non-significant result, p = .1, and observed power greater than .50. Power should equal 50% when the observed test-statistic matches the criterion.

Aside from this specific problem, observed power based on an observed effect size is problematic because effect sizes can vary widely due to sampling error. It is better to examine power for a theoretically interesting a priori effect size (e.g., d = .2 small effect, or d = .5 moderate effect).

https://replicationindex.wordpress.com/tag/observed-power/

Improve this answer
$\endgroup$
1
  • $\begingroup$ There is a lot of good information here. Note that post-hoc power is predicated on the observed effect size being correct. This is just an assumption to understand when you run such an analysis. $\endgroup$
    – gung - Reinstate Monica
    May 2, 2015 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.