0
$\begingroup$

I am trying to conduct power calculations for a "Before-After-Control-Impact" type of study focused on public opinion variables.

Ideally, I want to look at the relationship between effect size and power (in this case our sample size is a given). At a minimum, I want to be able to calculate the minimum detectable effect size for the conventional power = 0.80, alpha = 0.05 thresholds.

I am a bit flummoxed at the best approach to calculate power for a difference in difference of proportions. For example, in the case of "do you agree with Y?" I ultimately want to look at $(p_{ Treatment_{After}} - p_{Treatment_{Before}}) - (p_{Control_{After}} - p_{Control_{Before}})$. The most analogous thing I can think of would be an odds ratios approach, but I'm not confident this makes sense.

I realize this is a simple question, but most of what I can find is about comparing two proportions, rather than comparing the difference between two proportions.

$\endgroup$
2
  • $\begingroup$ After some searching, I think what I am trying to do is better stated as a sensitivity analysis (focused on effect size) for two (differences in) proportions in a repeated measures design. $\endgroup$ Feb 18 at 14:49
  • 1
    $\begingroup$ I've answered the main question below but repeated measures of the same respondents (i.e. if you're doing a panel survey) introduces some additional complexities. You might start by looking at this article: ncbi.nlm.nih.gov/pmc/articles/PMC6663085 $\endgroup$
    – num_39
    Feb 19 at 6:43

1 Answer 1

2
$\begingroup$

To calculate power for the difference in proportions between two populations, you need to find the standard error of this difference: $\sqrt{p_1 (1 - p_1) / n_1 + p_2 (1 - p_2) / n_2}$

To make this simple, let's assume equal sample size and variance. A little algebra gives us the following conservative bound for the standard error: $ se = 1 / \sqrt(n)$.

Let's assume an effect size of 0.1. Now for power of 80 percent and a significance level of 0.05, you need an effect that's 2.8 standard errors from zero. This gives us the following equation: $2.8 \times 1/ \sqrt{n} = 0.1$ A little algebra gives us: $n = (2.8 / 0.1)^2$ So utilizing this conservative bound on the variance, to detect an effect size 80 percent of the time at the significance level of 0.05, you'd need a sample size of 784 (392 for each population).

Now let's extend this to the case of a difference in the difference between two populations. Again, assuming equal sample sizes and variance, we have:

$$\sqrt{p(1 - p) / (n/4) + p(1 - p) / (n/4) +p(1 - p) / (n/4) +p(1 - p) / (n/4)}$$

This simplifies to $2 / \sqrt{n}$.

So if we replace the standard error for the difference between two proportions with the standard error of the difference in the difference between two proportions, we have: $n = (2.8 \times 2 ) / 0.1)^2$ again assuming an effect size of 0.1. This gives us a required sample of 3136 meaning that for the first survey, you'd need a sample of 1568 (784 for each population) and for the second survey, you'd also need a sample of 1568 (784 for each population).

For a very nice explanation of sample size, see chapter 16 of Regression and Other Stories by Gelmen, Hill, and Vehtari (available on line for free). For the case of difference-in-difference, look at the section on interactions.

$\endgroup$
3
  • $\begingroup$ Thanks! I'm marking this answer as accepted because it is thorough, involves great references, and is better than where I had gotten to by myself. That said, 2 follow up questions: 1) Doesn't having an effect size of anything violate the assumption of equal variance? 2) Would this difference in difference power be the power = 0.80 to detect differences between the two sampled populations longitudinally or the power = 0.80 to detect any differences between the 4 proportions? $\endgroup$ Feb 22 at 18:41
  • 1
    $\begingroup$ 1) Yes, but this is a conservative bound utilizing a proportion of 0.5 which gives us the highest possible variance for a proportion. If you replace p with the actual expected proportions for each sample, you'll have a slightly smaller require sample size. $\endgroup$
    – num_39
    Feb 23 at 4:09
  • 1
    $\begingroup$ 2) Not sure exactly what you're asking here but it's the power to detect the difference in the differences: $ (T_{t1} - T_{t2}) - (C_{t1} - C_{t2})$. If you simply had four samples and wanted to see if one of them was significantly different from the others, you'd want to use a different approach. $\endgroup$
    – num_39
    Feb 23 at 4:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.