4
$\begingroup$

Per reviewer request, I need to do power analysis for a logistic regression model with multiple dummy variables. I have four groups: Control, (Treatment) A, B, and C. The hypothesis is that group A and B do NOT differ from the control group, but group C does. I tested this hypothesis by running a logistic regression model with 3 dummy variables with the control group being the baseline group, along with 3 control variables.

The results supported the hypothesis, but one of the reviewers recommended conducting power analysis, saying that the non-significant differences between control and group A, and between control and group B might be a result of lack of power.

I tried to conduct power analysis using G*Power, but because there are no true values of Pr(Y=1|X=1)H0 and Pr(Y=1|X=1)H1 as I'm dealing with political perception as the dependent variable, I couldn't do the analysis.

I found that power analysis simulation using R could be a solution, but unfortunately I don't have enough knowledge about R to adapt the code provide here (Simulation of logistic regression power analysis - designed experiments) to my study design.

Any help or suggestions would be greatly appreciated.

p.s.) I am just wondering what follows is a valid alternative method: - A meta-analysis study suggested that the mean correlation of the effect in question is about 5. - Use the value to calculate R2 and use the R2 value to calculate Cohen's f2. - Estimate effect size (i.e., small, medium, large) based on the f2 value. - If, for example, the effect size is large, plug .40 in the "Effect size f" box in G*Power and select ANCOVA to calculate power.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
5
$\begingroup$

I don't mean to sound flippant, but a post hoc power analysis is really useless at this point (see here or here or I think chapter 3 or 4 in this book).

EDIT: That book I linked to references this paper. The abstract reads, and I quote:

It is well known that statistical power calculations can be valuable in planning an experiment. There is also a large literature advocating that power calculations be made when- ever one performs a statistical test of a hypothesis and one obtains a statistically nonsignicant result. Advocates of such post-experiment power calculations claim the calculations should be used to aid in the interpretation of the experimental results. This approach, which appears in various forms, is fundamentally flawed. We document that the problem is extensive and present arguments to demonstrate the flaw in the logic. (Emphasis added)

You should be able to find a few other papers to support your claim that a post hoc power calculation doesn't tell you anything the p-value doesn't.

My recommendation is that you respond to the reviewer in a respectful way saying that this statistical literature has definitively said that a post hoc power calculation is not appropriate. Instead, you should report and interpret the 95% confidence intervals (see section 3.7 of the book I have referenced for tips on reporting non-significant results). Reference some actual academic articles, not blog posts as I have.

Improve this answer
$\endgroup$
1
2
$\begingroup$

This is a case where a power analysis may make sense to address design concerns.

The most efficient way to do power analysis for a complex non standard setting is usually by simulation. It is not clear what your problem with specifying true values is - substitute "assumed true values" if that makes your more comfortable.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ How does a post hoc power analysis make sense here, in the light of the well known problems with the entire concept? I do not see how there are design concerns here, nor how a post hoc power analysis would help in addressing them. $\endgroup$
    – Stephan Kolassa
    Commented Dec 21, 2018 at 13:29
  • 1
    $\begingroup$ @StephanKolassa Obviously taking an observed difference is not what you should do, it's thinking about plausible/minimal important differences and looking at whether you had power against them. Obviously, you should really do that when designing the experiment rather than afterwards, but it helps with interpretation (you found no significant difference, but under very plausible assumptions you had really low power to do so, in any case, so that does not say much). Much of the same purpose is of course served by confidence intervals. $\endgroup$
    – Björn
    Commented Dec 21, 2018 at 13:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.