Multiplicity in sample size calculation for stratified estimation problem

Question

We want to determine the public opinion about a recently administered intervention in a health care network. There's a brief questionnaire of 10 Y/N questions in which we'd like to estimate proportions within a specific margin of error (+/- 3%). Based on historical data, we feel we can reliably estimate response probabilities within specific groups. We're interested in stratified sampling in which we will use Horvitz-Thompson weighting to obtain the estimate of the population proportions of positive responses to the questions.

My question is: how does one account for the multiplicity in calculation of the margin of error for individual questions? Intuiviely, estimation leads to the same issues in multiplicity that inference does. We might expect that this specific survey obtained, for least one question, an estimate of the population proportion and its estimated standard error which is inconsistent with that of the actual long-run average one would obtain from an infinite number of independent replications of the experiment. Hence, you would like a more conservative estimate of the required sample size to achieve a "95% confidence level" uniformly across all questions, or that there is a "grand" margin of error of 5% or so.

Is this consistent with any existing survey methodology? Is multiple testing strictly considered only in the world of formal inference and p-values? How exactly would one go about calculating sample sizes or power with any existing multiplicity adjustment for estimation?

Answer 1

The issue of multiplicity in the context of confidence intervals is regrettably less popular that multiplicity in hypothesis testing while the same problems exist. In particular, if you want to control the probability of all your parameters being covered ("simultaneous" coverage) , you cannot use the standard "nominal" confidence intervals. Just like in hypothesis testing, you need to decide what is the measure of error you wish to control. If it is the probability of simultaneous coverage of all parameters (FWE equivalent) or the "non coverage rate" (FDR equivalent).

For simultaneous coverage, you could inflate the confidence level a-la Bonferronni. As usual, these might be needlessly conservative. To the best of my knowledge, not all multiple comparison procedures can be inverted to give confidence regions. Moreover, I am unfamiliar with any introductory text to the matter except Hochberg's and Tamahane's.

For non-coverage rate control things are easier as the nominal confidence intervals trivially guarantee an expected non coverage of $1-\alpha$ within and between samples.