3
$\begingroup$

We recently submitted a manuscript for publication. My advisors never do power analyses, and so the reviewers came back questioning our sample sizes. Lesson learned on my end. One reviewer said "a power analysis (e.g. sensitivity analysis)" is needed. I am not familiar with post hoc power analyses, and I have read the traditional post hoc test found in GPower is really not sufficient or helpful. I have been looking at the sensitivity test in GPower since that is what our reviewer recommended, but I am reading that may not be appropriate either? I would really love some insight on this so I can 1) resubmit our paper, 2) learn for the future.

We have a sample size of 45 and conduct a 3x(7) ANOVA. But we also further break it down to do 3 separate 2x2 ANOVAs on each age group, and also do a few paired t-tests for each age group looking at early vs. late task accuracy. The reviewers are concerned about these analyses in which the groups of 15 are tested separately (2x2's and t-test).

I have read that the sensitivity tests give you the "minimum important effect size for their research question". Which sounds promising! Is it really as simple as entering my t-test information into G*Power and getting the effect size and comparing if my effect sizes are greater? I'm seeing such mixed results online.

Thanks for you help.

$\endgroup$
2
  • $\begingroup$ Can you make your question a bit more explicit? I'm not sure exactly what you're asking. Perhaps give an example? $\endgroup$ Oct 6, 2023 at 5:43
  • 3
    $\begingroup$ I like the chapter on Sample Size Justification in Improving Your Statistical Inferences by Daniël Lakens. It has a section "What to do if Your Editor Asks for Post-hoc Power?" which is the situation you are facing. $\endgroup$
    – dipetkov
    Oct 6, 2023 at 7:24

1 Answer 1

3
$\begingroup$

This is not a direct answer to you question but I hope it will help you anyway.

It seems that the reviewer has misunderstood something. Power analyses and a sensitivity analyses have nothing to do with each other. In fact they serve two completely different purposes.

A power analysis is very helpful in planning a study when you need to get an idea of how many subjects you need to enroll, but as described here and here it does not make any sense to do a post-hoc power analysis or to use the study results to calculate a minimum important effect size. If the reviewers fear that your study is under powered all they have to do is to look at the width of the confidence intervals (CI). The first link above has some excellent references that you can use to "push back" on the reviewers

Sensitivity analyses, on the other hand, are used to assess the robustness of the findings from your primary analyses, and are always reasonable to include. They are done to assess the impact of the key assumptions underlying the model you have used to analyze your data on your conclusions. See for example here and here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.