I have a retrospective study of a medical procedure (n=~150) resulting in a binary pass (2/3rds)/fail (1/3rd) outcome. T-Test/Mann-Whitney were performed on continous variables with Chi-Square and Fisher exact tests performed on binary/classifiers, determining statistical significance for p values <0.05, bonferroni correction was not used to determine significance level.
A reviewer has asked to perform power analysis, determining whether the study is sufficiently powered to show an actual effect. It is my understanding this is an incorrect approach:
- Power analysis should be performed prior to the study, and it's value should be determined based upon estimated effects rather than observed, primarily to optimize sample sizes.
- Secondly, it is simply the the probability of a type II error, albeit practically in a post-hoc setting (using observed effects) the power analysis will simply be the inverse of the p-value calculated for each variable.
- Given the mast majority of variables were determined to be non-significant and therefore not-reject the null, the power (or median power) of the study will be lower.
What would you suggest to be a better alternative, I could simply calculate given the sample size what the power would have been for an effect of 0.3? Would it be better to demonstrate the power at each effect level for recommendation of future studies to confirm my findings? Just trying to understand what the reviewer is looking to understand from performing this analysis.