We have an experiment in which the minimum sample size was calculated, but due to the pandemic, data collection had to be terminated earlier, not reaching the minimum size. The statistical analysis was done, but the person responsible for the project is concerned with the article. It was suggested that you do a post hoc power analysis, showing how much power was lost with the early closure of the data collection, but I saw that doing this analysis is not recommended (Hoenig & Heisey, 2001). So, how should I proceed? Is there an alternative to post hoc power analysis? How should I report/justify about the sample size being smaller than calculated?
2
-
$\begingroup$ Did you get a significant result? $\endgroup$– DaveFeb 16, 2021 at 19:21
-
$\begingroup$ Wouldn't you quantify the uncertainty, the % confidence rating? How was the design minimum sample size originally calculated? If you went back to those calculations, and plugged in your actual sample size, what confidence rating remains? (on a related note, my wife tells me engineers are nerds. I keep telling her, that her sample size is too small... :^) $\endgroup$– zipzitFeb 17, 2021 at 3:24
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Elaborating a bit on Jeremy's answer, let's think for a minute about what a power analysis is. The purpose is to determine how many participants one would need to "detect" an effect of a specific size. So in discussing the results of your experiment vis a vis the sample size you originally designed, and what the pandemic (unforseen circumstances) led to, you should do so in the context of the estimated effect size of your current experiment. Is the effect size about what was expected a priori? Bigger? Smaller? Would a larger sample size potentially have changed the results of the hypothesis test(s) of this particular effect? That kind of discussion is what is called for, since you're being honest about why you ended up with the N that you did. Then you're contextualizing the resulting hypothesis tests through the lens of effect sizes.
$\begingroup$
$\endgroup$
1
I would just explain what happened. You powered for N, and you got N*. It's not the first time this has happened (and won't be the last).
Post hoc power would not be especially useful (as you have realized.)
answered Feb 16, 2021 at 18:51
-
5$\begingroup$ I think post hoc power could be a good addition here, provided they use the a-priori effect size, not the observed effect size. $\endgroup$ Feb 16, 2021 at 19:00