# Calculate power-effect size relationship for difference in difference of proportions

I am trying to conduct power calculations for a "Before-After-Control-Impact" type of study focused on public opinion variables.

Ideally, I want to look at the relationship between effect size and power (in this case our sample size is a given). At a minimum, I want to be able to calculate the minimum detectable effect size for the conventional power = 0.80, alpha = 0.05 thresholds.

I am a bit flummoxed at the best approach to calculate power for a difference in difference of proportions. For example, in the case of "do you agree with Y?" I ultimately want to look at $$(p_{ Treatment_{After}} - p_{Treatment_{Before}}) - (p_{Control_{After}} - p_{Control_{Before}})$$. The most analogous thing I can think of would be an odds ratios approach, but I'm not confident this makes sense.

I realize this is a simple question, but most of what I can find is about comparing two proportions, rather than comparing the difference between two proportions.

• After some searching, I think what I am trying to do is better stated as a sensitivity analysis (focused on effect size) for two (differences in) proportions in a repeated measures design. Feb 18 at 14:49
• I've answered the main question below but repeated measures of the same respondents (i.e. if you're doing a panel survey) introduces some additional complexities. You might start by looking at this article: ncbi.nlm.nih.gov/pmc/articles/PMC6663085 Feb 19 at 6:43

To calculate power for the difference in proportions between two populations, you need to find the standard error of this difference: $$\sqrt{p_1 (1 - p_1) / n_1 + p_2 (1 - p_2) / n_2}$$

To make this simple, let's assume equal sample size and variance. A little algebra gives us the following conservative bound for the standard error: $$se = 1 / \sqrt(n)$$.

Let's assume an effect size of 0.1. Now for power of 80 percent and a significance level of 0.05, you need an effect that's 2.8 standard errors from zero. This gives us the following equation: $$2.8 \times 1/ \sqrt{n} = 0.1$$ A little algebra gives us: $$n = (2.8 / 0.1)^2$$ So utilizing this conservative bound on the variance, to detect an effect size 80 percent of the time at the significance level of 0.05, you'd need a sample size of 784 (392 for each population).

Now let's extend this to the case of a difference in the difference between two populations. Again, assuming equal sample sizes and variance, we have:

$$\sqrt{p(1 - p) / (n/4) + p(1 - p) / (n/4) +p(1 - p) / (n/4) +p(1 - p) / (n/4)}$$

This simplifies to $$2 / \sqrt{n}$$.

So if we replace the standard error for the difference between two proportions with the standard error of the difference in the difference between two proportions, we have: $$n = (2.8 \times 2 ) / 0.1)^2$$ again assuming an effect size of 0.1. This gives us a required sample of 3136 meaning that for the first survey, you'd need a sample of 1568 (784 for each population) and for the second survey, you'd also need a sample of 1568 (784 for each population).

For a very nice explanation of sample size, see chapter 16 of Regression and Other Stories by Gelmen, Hill, and Vehtari (available on line for free). For the case of difference-in-difference, look at the section on interactions.

• Thanks! I'm marking this answer as accepted because it is thorough, involves great references, and is better than where I had gotten to by myself. That said, 2 follow up questions: 1) Doesn't having an effect size of anything violate the assumption of equal variance? 2) Would this difference in difference power be the power = 0.80 to detect differences between the two sampled populations longitudinally or the power = 0.80 to detect any differences between the 4 proportions? Feb 22 at 18:41
• 1) Yes, but this is a conservative bound utilizing a proportion of 0.5 which gives us the highest possible variance for a proportion. If you replace p with the actual expected proportions for each sample, you'll have a slightly smaller require sample size. Feb 23 at 4:09
• 2) Not sure exactly what you're asking here but it's the power to detect the difference in the differences: $(T_{t1} - T_{t2}) - (C_{t1} - C_{t2})$. If you simply had four samples and wanted to see if one of them was significantly different from the others, you'd want to use a different approach. Feb 23 at 4:19