2
$\begingroup$

I am used to performing power analysis with unweighted data. In R, I use the pwr package, which is based on Cohen (1988).

How would you do power analysis on weighted survey data? I don't think you could use the same calculations. The weights would probably somehow have to figure into it? Is there a reference that discusses this?

$\endgroup$

2 Answers 2

3
$\begingroup$

The weights do figure in a major way. The design effect (the ratio of variances for the complex design and an SRS that you are used to power-analyzing) due to unequal weights is customarily written as $$ {\rm DEFF}_w = \frac{V_{\rm Design}[y]}{V_{\rm SRS}[y]}=1+{\rm CV}^2_w = \frac{n\sum_i w_i^2}{\bigl(\sum_i w_i)^2} $$ So as a very first approximation, you can compute the expected CV of weights for your study, compute this DEFF, and pro-rate the sample size computed from the simple unweighted analysis by that number. This procedure assumes that the weights are not correlated with your important analysis variables (the outcome itself, the treatment groups, etc.), which may or may not be a reasonable assumption. A more accurate analysis would have to take stratification and other complex sample features explicitly.

As Greg Snow correctly pointed out, a simulation study would indeed be the next logical step in trying to do this with an utmost precision. Doing simulations with unequal probabilities of selection, however, is a fairly complicated business, and if you are new to it, you will likely do it wrong for the first five or so times.

$\endgroup$
6
  • 1
    $\begingroup$ Sorry to resurrect an old post. What do you mean by "pro-rate" the sample size computed from the simple unweighted power analysis? I'm trying to determine the needed sample size for a chi square using complex stratified data. Does that mean I should do a simple chi-square sample size analysis, then adjust the required sample size with the DEFF? How shall I do that? $\endgroup$
    – JJM
    Commented Oct 15, 2014 at 14:26
  • $\begingroup$ Your best bet would probably be a simulation. This DEFF is only applicable when the unequal weights is the only complication. With a more complex design, you are looking at generalized design effects and Rao-Scott corrections and what not, and it will likely get too messy to be analytically tractable. $\endgroup$
    – StasK
    Commented Oct 15, 2014 at 16:05
  • 1
    $\begingroup$ I can't find any resources through Google or CV that explain how to do a power analysis by simulation of complex survey data. Do you know of any online resources, or are textbooks my only hope? $\endgroup$
    – JJM
    Commented Oct 15, 2014 at 18:21
  • $\begingroup$ If it is not touched upon in the Practical Tools book, then you are on your own. $\endgroup$
    – StasK
    Commented Oct 15, 2014 at 20:42
  • 1
    $\begingroup$ @AyalewA., Korn & Graubard (1999) book talks about this. $\endgroup$
    – StasK
    Commented Jan 30, 2015 at 15:50
1
$\begingroup$

For power studies beyond the simple 2 sample t-test it is probably best to use simulations to estimate the power.

This Answer gives more detail and an example (though not of weighted regression, but the idea would be similar).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.