Power analysis: difference between observed and expected effect size

Question

Im trying to understand the ramifications of an observed effect size being much smaller than the expected effect size used in an a priori power analysis

Here's a hypothetical situation...

Lets say I run a power analysis expecting a standardized effect size of 0.2 (for a beta in an OLS regression, if that matters). The 0.2 is just an estimate and is the best value I can come up with given limited research in this area. The analysis tells me I need 200 people to detect an effect of that size with 80% power.

Then I collect my data (more than the required 200) and run the analysis. It turns out that none of the predictors in my regression model are statistically significant, and also that the observed effect size is actually much smaller, closer to 0.01.

A colleague/supervisor/reviewer claims that my non-significant effects are due to being underpowered. In other words, I planned for 200 people due to power analysis results, but I used an effect size estimate that was much larger than what I actually observe in my data i.e. I overestimated the size of my effect.

Is there any validity to their claim? and what could I say in defense of my results?

If you use the best possible estimate of an effect size but observe much smaller (non-significant) effects, is it still possible for null results to be "underpowered"?

To me, this feels like sample size justification by post-hoc power analysis, but in reverse. That statement is using observed data to make claims about relationships in the population, if that makes sense...?

Answer 1

Firstly, you do not "detect an effect of that size with 80% power", you have 80% power to have a statistically significant result assuming the true effect size is exactly what you assumed (plus the other assumptions e.g. about variability need to be right). When you get a significant result the effect size will not be exactly what you assumed and will in fact be biased upwards. Across all possible realizations of a study, it will be bigger than the true effect 50% of the time and larger 50% of the time.

Secondly, you could indeed be underpowered for a smaller effect size (there is always some effect size for which a study would be poorly powered). If the effect sizes for which you are still underpowered are of practical relevance, this may be an issue from a certain point of view (the experiment is not really that useful, the conclusion is essentially that we cannot exclude that there might be an important effect, but we have not shown that it exists). If your experiment more or less excluded any effect size of practical importance, most people would probably agree that that's just fine.