2
$\begingroup$

Im trying to understand the ramifications of an observed effect size being much smaller than the expected effect size used in an a priori power analysis

Here's a hypothetical situation...

Lets say I run a power analysis expecting a standardized effect size of 0.2 (for a beta in an OLS regression, if that matters). The 0.2 is just an estimate and is the best value I can come up with given limited research in this area. The analysis tells me I need 200 people to detect an effect of that size with 80% power.

Then I collect my data (more than the required 200) and run the analysis. It turns out that none of the predictors in my regression model are statistically significant, and also that the observed effect size is actually much smaller, closer to 0.01.

A colleague/supervisor/reviewer claims that my non-significant effects are due to being underpowered. In other words, I planned for 200 people due to power analysis results, but I used an effect size estimate that was much larger than what I actually observe in my data i.e. I overestimated the size of my effect.

Is there any validity to their claim? and what could I say in defense of my results?

If you use the best possible estimate of an effect size but observe much smaller (non-significant) effects, is it still possible for null results to be "underpowered"?

To me, this feels like sample size justification by post-hoc power analysis, but in reverse. That statement is using observed data to make claims about relationships in the population, if that makes sense...?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Firstly, you do not "detect an effect of that size with 80% power", you have 80% power to have a statistically significant result assuming the true effect size is exactly what you assumed (plus the other assumptions e.g. about variability need to be right). When you get a significant result the effect size will not be exactly what you assumed and will in fact be biased upwards. Across all possible realizations of a study, it will be bigger than the true effect 50% of the time and larger 50% of the time.

Secondly, you could indeed be underpowered for a smaller effect size (there is always some effect size for which a study would be poorly powered). If the effect sizes for which you are still underpowered are of practical relevance, this may be an issue from a certain point of view (the experiment is not really that useful, the conclusion is essentially that we cannot exclude that there might be an important effect, but we have not shown that it exists). If your experiment more or less excluded any effect size of practical importance, most people would probably agree that that's just fine.

Improve this answer
$\endgroup$
4
  • $\begingroup$ But how could you be sure that the observed data is a case of a smaller effect size? Couldnt it be possible that the observed effect is small due to sampling issues (or something else), rather than being underpowered, given youre only looking at a sample of a population? $\endgroup$
    – Simon
    May 9, 2018 at 21:46
  • $\begingroup$ You indeed can't be sure of that. If no biases/sampling issues are present, then you can at least quantify the standard error and form confidence intervals (i.e. quantify how much sampling error there might be). $\endgroup$
    – Björn
    May 10, 2018 at 6:00
  • $\begingroup$ good point about the CIs. If my planned sample size (0.20) falls within the range of the calculated CI, would it be safe to say their concern carries less weight as we cant confidently reject 0.20 as a possible size for the effect? $\endgroup$
    – Simon
    May 10, 2018 at 18:06
  • $\begingroup$ Well, the data would also seem pretty consistent with any effect size between that and no effect. That your target effect size is in the CI may just bee a function of you not having all that much data. $\endgroup$
    – Björn
    May 10, 2018 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.