Im trying to understand the ramifications of an observed effect size being much smaller than the expected effect size used in an a priori power analysis
Here's a hypothetical situation...
Lets say I run a power analysis expecting a standardized effect size of 0.2
(for a beta in an OLS regression, if that matters). The 0.2
is just an estimate and is the best value I can come up with given limited research in this area. The analysis tells me I need 200 people to detect an effect of that size with 80% power.
Then I collect my data (more than the required 200) and run the analysis. It turns out that none of the predictors in my regression model are statistically significant, and also that the observed effect size is actually much smaller, closer to 0.01
.
A colleague/supervisor/reviewer claims that my non-significant effects are due to being underpowered. In other words, I planned for 200 people due to power analysis results, but I used an effect size estimate that was much larger than what I actually observe in my data i.e. I overestimated the size of my effect.
Is there any validity to their claim? and what could I say in defense of my results?
If you use the best possible estimate of an effect size but observe much smaller (non-significant) effects, is it still possible for null results to be "underpowered"?
To me, this feels like sample size justification by post-hoc power analysis, but in reverse. That statement is using observed data to make claims about relationships in the population, if that makes sense...?