Report power of 4-way repeated measures ANOVA?

Question

I'm a french psychologist (so, not a statistician :-))

I have just submit an article and the reviewers respond us:

One should report power, which directly indexes the probability of making a Type II error. Why? Because some of your conclusions are based on not obtaining a significant interaction, you want to ensure that the reason for this null interaction is because there is truly not an interaction. The other (bad) possibility is that you didn't find a true interaction because your sample size was too small (which could happen, especially if the interaction is theoretically pretty small). The way to argue against this is to report power. Power can be output as an option in SPSS, or one can use free software such as GPower (http://www.gpower.hhu.de/en.html) or MorePower (https://wiki.usask.ca/display/MorePowerCalculatorV6/Home)*

I understand his comment but how to do this with 4-way repeated measures ANOVA? It is also important to know that we already reported the eta-square (usually found in such study -experimental psychology-) as a measure of effect size when running ANOVA (with .02 =small effect, .13 =medium effect and .26 =large effect). Apparently, that does not seem to be enough...

Furthermore, I have red a lot of comments saying that it is not relevant to report post-hoc power: the usefulness of retrospective techniques is controversial. Falling for the temptation to use the statistical analysis of the collected data to estimate the power will result in uninformative and misleading values. In particular, it has been shown.

In your opinion, what should be answered?

Thank you!

Answer 1

There are really two questions here.

First, is the request reasonable? There have been lots of discussions of post-hoc power analysis, both here and elsewhere. My view is that it should not be done, but .... you have to satisfy the journal editors, not me.

Second, if you are going to do it, how should you do it? With such a complex design, I think the only way to get a really good power estimate is going to be through simulation. Simulation for power analysis has also been discussed here. Exactly how to simulate will depend on many things, but will include the program you have available.