Does a post-hoc power analysis suffice in a psychological paper?

Question

I am a psychologist and I have the following question:

Last year I was running a study. I intended to collect at least 400 participants (the anticipated effect size was hard to estimate). The study was an experiment with 6 conditions (2 (self vs other) x 3 (situation with 3 different levels)) and I was interested in a 3-way interaction (self_vs_other x situation x personality trait) in a multiple hierarchical regression.

My data collection got terminated midway because the Institute closed the surveymonkey account. I ended up with 236 participants. I calculated results and found what I was looking for. The interaction alone explains 4.3% of variance and is of size eta_squared =.094 which means it's somewhere between medium and large. Based on post-hoc analysis, I had 89% power to detect this effect.

But I am unsure about sample size. Someone might say it's too small for a study with a 3-way interaction.

Now, can I simply report the post-hoc power analysis in a paper and claim that it's ok? Or should I rather resume collection (on a different site and in a different situation: during covid pandemic...)?

Thank you all in advance for your input.

Answer 1

The way you describe it, i.e. estimating the effect size based on the data and then checking post-hoc the power based on the estimated effect size is how post-hoc tests are often done. Unfortunately, it is NOT how post-hoc power estimates should be done. See for example O'Keefe (2010).

The reason for this is, that a post-hoc power analysis of this type never gives any useful information. If the effect is not significant, then the study is always underpowered to detect an effect of the given size (otherwise it would have been significant). Similarly, the post-hoc power is not interesting when the result is significant.

In essence, post-hoc power should be done in a similar fashion as a-prioi power. That is, it should be based on possible population effect sizes, not on sample effect sizes estimated in the study. That is, you should either analyze it based on effect sizes in the literature and see if you could have detected those. This is especially informative if the estimated effect size in your study is larger than the one reported in the literature.

About your question, if post-hoc power is sufficient for a psychological publication:

• Most psychological publications still have no power analysis at all. While that is definitely bad practice, it shows you still have good chances even without any kind of power analysis.
• It seems you did a power analysis of some kind, which led to the estimated 400 participants required. You can report that. If you did not reach that number due to reasons that were beyond your control, then that is a valid reason to reduce the sample size. You could report that and hope the reviewer accepts it.
• The main reason why one should reach the desired number of participants after a power analysis (and not more or less), is to avoid optional stopping, which is a bad case of p-hacking. If you can show sufficiently, that there is no p-hacking in your analysis, then that would likely also suffice. Thus, report your power analysis, and then indicate why you stopped earlier.

NOTE: Concerning the last two points, it is easy to fool oneself, and still engage in p-hacking without noticing it. You should ask yourself the question (and be sincere to yourself): "If I had not gotten a significant result, would I have tried to get additional participants until I reached the N=400." If you can't sincerely answer this question with "NO", then you are actually p-hacking. You just happened to be lucky with it.