Sample size calculation for a difference between two groups with clustered data

Question

I want to compare proportions or means in two groups in a clustered sample. Suppose the sampled clusters are schools or hospitals or Census areas, and we sample from each of these an equal number in each of the two comparison groups. In other words, it is not the case that entire clusters are in one comparison group or another (i.e., not a cluster randomized trial).

In survey analysis, standard errors for estimating means and proportions are typically inflated in cluster samples because of the intraclass correlation (ICC) among subjects in clusters -- we sample $n$ subjects but we really have a smaller effective sample size due to the ICC/design effect.

However, when we want to compare two groups in a clustered sample, and both groups are present in all clusters, my simulations for various ICC values are basically finding that the various approaches for estimating a difference between the two groups (in particular, survey regression and GEE with an exchangeable working correlation matrix) have roughly similar standard errors to the standard error associated with a t-test that ignores the clustering, on average. In fact, on average, the standard error is marginally smaller as compared to the approach that ignores the clustering in the data.

My first question is: Is this correct? Does it make sense? Have I made some mistake?

Intuitively, I think it makes some sense to me -- we can think of observations in the two groups as almost forming pairs in each cluster, and paired t-tests typically have greater power than two-sample independent t-tests. It seems this extra statistical power from having both groups present in each cluster can compensate for the increased uncertainty present in each group's respective estimates.

I found some literature that seems to agree with me on this, but it's not much: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-9-39 "Thus, in an observational study, with all centers having identical group distributions [. . .] taking into account the center effect leads to increased power"

For SRS, the variance for a difference in means is: $Var(\hat{y}_{1, srs} - \hat{y}_{2, srs}) = Var(\hat{y}_{1, srs}) + Var(\hat{y}_{2, srs})$

For a clustered sample, the variance for a difference in means is: $Var(\hat{y}_{1, cluster} - \hat{y}_{2, cluster}) = Var(\hat{y}_{1, cluster}) + Var(\hat{y}_{2, cluster}) - 2Cov(\hat{y}_{1, cluster}, \hat{y}_{2, cluster})$

In other words, it can be the case that despite $Var(\hat{y}_{i, srs}) < Var(\hat{y}_{i, cluster})$ for $i = 1, 2$, positive covariance between the two cluster estimates can potentially actually more than compensate. Is this correct? Or is there some quirk in my simulations causing this to happen?

My second question is: are there any suggestions on sample size calculations for such a study to ensure it is sufficiently powered? Simulations like I have been doing appear to be the best solution, but I'm wondering if I'm missing something obvious, and I do not want to under-power the study. I see there is a package in R called samplesize4surveys with functions for differences in means and proportions, but I have trouble parsing some of the documentation and replicating simple SRS results. I would appreciate any references as well.

In practice my outcome of interest is binary, not continuous, but I think the same lessons should hold.

Answer 1

I provide a partial answer to my own question as I have researched a bit more, in case others would benefit from it -- but anyone who can add in their own expertise would be welcome.

1. A look into the literature about this issue.

From Leslie Kish's Survey Sampling, page 582, section, "How to present standard errors of comparisons?" he writes:

"Differences between means of similar characteristics play crucial roles in survey analysis; sometimes for comparing two separate samples, but most frequently for comparing two subclasses in the same sample. [. . .] However, two subclass means from clustered samples, although based on distinct sets of elements, tend to come from the same set of clusters. The positive correlation between cluster influences on the two means tends to reduce the variance of the difference. It seems that for most comparisons the variance is reduced below the sum of the individual variances, but probably not below the variance of two simple random samples" (emphasis mine)

Of note, on page 581, Table 14.1.IV contains design effects for proportions and design effects for differences in proportions from successive surveys that had the same clusters (primary sampling units) but not the same units/households. It shows mostly design effects lower for the difference than the proportion, and even for some lower proportions, the design effect is less than one. The table's caption notes, "The correlations reduce the design effect on the variance of the differences [. . .] The buying intentions (10-16) have low percentages, and they show low values of $\sqrt{Deff}$. This occurs often with rare characteristics." I have not read Kish in great detail but there may be more there. There is also Table 14.1.III ("Example of sampling errors of differences between percentages") but it doesn't make much sense to me.

2. Some notes on my initial simulation results showing equivalent results to SRS (design effect = 1) for a difference in means.

If one simulates data where the covariance matrix was exchangeable -- that is, elements in group A and group B have the same covariance as elements in group A with each other and group B with each other ($Cov(Y_i, Y_j) = \sigma_{cluster}^2$ for all $i \neq j$, and $Cov(Y_i, Y_i) = Var(Y_i) = \sigma_{resid}^2 + \sigma_{cluster}^2$ for all $i$) -- you do obtain the result that the GEE or survey-adjusted results effectively have a very similar standard error to SRS data. However, this is sensitive to this assumption about what the covariance matrix looks like. In practice, elements from group A are likely more correlated with each other than they are with elements in group B, but there is still some correlation between the two groups within the cluster that is greater than 0. In this setting, the design effect of the difference is less than the individual mean or proportion design effects, but it is still likely greater than 1.

I also found an example in Heeringa et al's Applied Survey Data Analysis (1st edition, Chapter 5.6.1), in which a difference in means in two subgroups had a larger standard error than one would expect reading the above from Kish -- apparently there was an small but negative sample covariance between the two group estimates.