# Power analysis with weighted survey data

I am used to performing power analysis with unweighted data. In R, I use the pwr package, which is based on Cohen (1988).

How would you do power analysis on weighted survey data? I don't think you could use the same calculations. The weights would probably somehow have to figure into it? Is there a reference that discusses this?

The weights do figure in a major way. The design effect (the ratio of variances for the complex design and an SRS that you are used to power-analyzing) due to unequal weights is customarily written as $${\rm DEFF}_w = \frac{V_{\rm Design}[y]}{V_{\rm SRS}[y]}=1+{\rm CV}^2_w = \frac{n\sum_i w_i^2}{\bigl(\sum_i w_i)^2}$$ So as a very first approximation, you can compute the expected CV of weights for your study, compute this DEFF, and pro-rate the sample size computed from the simple unweighted analysis by that number. This procedure assumes that the weights are not correlated with your important analysis variables (the outcome itself, the treatment groups, etc.), which may or may not be a reasonable assumption. A more accurate analysis would have to take stratification and other complex sample features explicitly.

As Greg Snow correctly pointed out, a simulation study would indeed be the next logical step in trying to do this with an utmost precision. Doing simulations with unequal probabilities of selection, however, is a fairly complicated business, and if you are new to it, you will likely do it wrong for the first five or so times.

• Sorry to resurrect an old post. What do you mean by "pro-rate" the sample size computed from the simple unweighted power analysis? I'm trying to determine the needed sample size for a chi square using complex stratified data. Does that mean I should do a simple chi-square sample size analysis, then adjust the required sample size with the DEFF? How shall I do that?
– JJM
Commented Oct 15, 2014 at 14:26
• Your best bet would probably be a simulation. This DEFF is only applicable when the unequal weights is the only complication. With a more complex design, you are looking at generalized design effects and Rao-Scott corrections and what not, and it will likely get too messy to be analytically tractable. Commented Oct 15, 2014 at 16:05
• I can't find any resources through Google or CV that explain how to do a power analysis by simulation of complex survey data. Do you know of any online resources, or are textbooks my only hope?
– JJM
Commented Oct 15, 2014 at 18:21
• If it is not touched upon in the Practical Tools book, then you are on your own. Commented Oct 15, 2014 at 20:42