2
$\begingroup$

I'm trying to learn about cluster-randomized trials, and I see that people use clustered-standard errors when the unit of analysis is different than the unit of treatment assignment. If not, the false positive rate for the test is higher than the type I error being set. Can you give me a mathematical proof of the 2 following statements?

  • Using a regular t-test on a cluster-randomized trial with a given alpha gives higher false positive rate.
  • Using a OLS with clustered-standard errors gives a FPR equal to the alpha.
$\endgroup$
2
  • $\begingroup$ See stats.stackexchange.com/a/554287/7071, especially the C&M paper. $\endgroup$
    – dimitriy
    Oct 6, 2023 at 8:58
  • $\begingroup$ Note that use of OLS with an after-the-fit correction to the SE doesn’t fix the suboptimality of OLS regression parameter estimates under the working independence model. Better to use a random effects model, with Bayesian random effects models working better than frequentist ones. Note that the unit of analysis is still the patient whether randomizing clusters or individuals. The effective sample size is somewhere between the number of clusters and number of patients. $\endgroup$ Oct 10, 2023 at 12:07

1 Answer 1

2
+50
$\begingroup$

This post addresses your first question.

Using a regular t-test on a cluster-randomized trial with a given alpha gives higher false positive rate.

This is not necessarily true.

The difference between a cluster v. non-clustered trial can be understood in terms of the variance of a cluster v. non-clustered sample mean. In practice, a clustered sample mean typically has higher variance than a non-clustered sample mean but this is not necessarily the case as you will see below.

I derive the variance for a one-stage clustered sample mean and compare this to the variance of non-clustered (SRS) sample mean for the case where clusters are of equal size.

Definitions

  • $N$ = number of primary sampling units (psus) in the population
  • $n$ = number of psus in the sample
  • $M$ = number of secondary sampling units (ssus) in each cluster where $M$ is fixed and identical for each cluster
  • $N \cdot M$ = the population
  • $\mu$ = unit mean
  • $\mu_c$ = cluster mean

Now consider a sample of size $n \cdot M$.

Simple Random Sample

Proposition 1.1

If we take a simple random sample (SRS) of the population. For simplicity, imagine a case where $n$ is negligible compared to $N$ such that the finite population correction is legible (or imagine a sample with replacement).

We know that the sample mean is an unbiased estimate of the population mean, and the derivation of the variance of the sample mean is as follows.

$$ \begin{aligned} V_{SRS}(\hat{\mu}) &= \frac{S^2}{n \cdot M} \\ \end{aligned} $$

where $S^2 = \frac{1}{NM - 1} \sum^{NM}_{i = 1} (y_i - \mu)^2$ We also know that the sample variance, $s_2$ is an unbiased estimate for the population variance, $S^2$.

Proofs for these claims can be found in any standard textbook on sampling.

One-stage cluster samples with clusters of equal sizes

Imagine instead we sample $n$ psus and enumerate each of $M$ ssus in those $n$ psus. Again, the sample average is an unbiased estimator of the population mean, $\mu$. I omit this proof.

Proposition 1.2

$$ V_{Cluster}(\hat{\mu}) = \frac{S_c^2}{n \cdot M^2} $$ where where $S^2_c = \frac{1}{N - 1} \sum^N_{j = 1} (\mu_j \cdot M - \mu_c)^2$. Recognize that $S^2_c$ is the variance of the cluster average (rather than the unit average).

Proof 1.2

We start with the derivation of the variance of the cluster average, $\mu_c$. Since clusters are sampled via a simple random sample with replacement, by Proposition 1.1, the variance is

$$ V_{Cluster}(\hat{\mu_c}) = \frac{S_c^2}{n} $$ From this, we can easily find the variance for the estimate of the population mean since $\mu_c / M = \mu$ and $\hat{\mu_j} / M = \hat{\mu}$.

$$ \begin{aligned} V_{Cluster}(\hat{\mu}) = V_{Cluster}(\frac{\hat{\mu_c}}{M}) &= \frac{1}{M^2} Var(\hat{\mu_c}) = \frac{S_c^2}{n \cdot M^2} \end{aligned} $$ This completes the proof.

In order to make comparisons between a cluster sample and a simple random sample, it's helpful to establish two alternative expressions of the variance of the mean for a one-stage cluster sample.

Correlary 1.0

$$ \begin{aligned} V_{Cluster}(\hat{\mu}) &= \frac{MSB}{n} \end{aligned} $$ where $MSB = \frac{1}{N - 1} \sum^N_{j = 1} (\mu_j - \mu)^2$.

Recall the variance for a simple random sample from Proposition 1.1.

$$ \begin{aligned} V_{SRS}(\hat{\mu}) &= \frac{S^2}{n \cdot M} \end{aligned} $$ Notice then that the difference between the variance for a one-stage cluster sample and a SRS reduces to the difference between $MSB$ and $\frac{S^2}{M}$. If, as is usually the case, the variance between clusters is greater than the population variance, then a cluster sample is less efficient, i.e. $V_{Cluster} > V_{SRS}$

Proof for Correlary 1.0

Start by expanding $S^2_c$.

$$ \begin{aligned} S^2_c &= \frac{1}{N - 1} \sum^N_{j = 1} (\mu_j \cdot M - \mu_c)^2 \\ &= \frac{1}{N - 1} \sum^N_{j = 1} (\mu_j \cdot M - \mu \cdot M)^2 \\ &= \frac{M^2}{N - 1} \sum^N_{j = 1} (\mu_j - \mu)^2 \\ &= M^2 \cdot MSB \end{aligned} $$ Substituting back into the original equation gives us

$$ \begin{aligned} V_{Cluster}(\hat{\mu}) &= \frac{M^2 \cdot MSB}{n \cdot M^2} \\ &= \frac{MSB}{n} \end{aligned} $$ This completes the proof.

Correlary 1.1

An alternative expression of the variance of the sample mean from a one-stage cluster sample is a follows.

$$ V_{cluster} \approx \frac{ S^2 }{n \cdot M} \cdot (1 + (M-1) \cdot \rho) $$ where $S^2$ is the population variance as defined previously and where $\rho$, the interclass correlation coefficient, is defined as

$$ \begin{aligned} \rho &= \frac{E(y_{j, i} - \mu)(y_{j, k} - \mu)}{E(y_{j, i} - \mu)^2} \\ &= \frac{2 \sum^N_{j = 1} \sum^M_{i < k} (y_{j, i} - \mu)(y_{j, k} - \mu)}{(M - 1)(NM - 1) S^2} \end{aligned} $$

Proof of Correlary 1.1

Starting again with the definition of $S^2_c = \frac{1}{N - 1} \sum^N_{j = 1} (\mu_j \cdot M - \mu_c)^2$, notice that

$$ \begin{aligned} (\mu_j \cdot M - \mu_c)^2 &= \left( \sum^M_{i = 1} y_i - \mu \right)^2 \\ &= \left( (y_1 - \mu) + (y_2 - \mu) + ... + (y_M - \mu)\right)^2 \\ &= \sum^M_{i = 1} (y_i - \mu)^2 + 2 \sum^M_{i > k}(y_i - \mu)(y_k - \mu) \end{aligned} $$ Adding the subscript for $j$ and plugging this into $S^2_c$ gives us

$$ \begin{aligned} S^2_c &= \frac{1}{N - 1} \sum^N_{j = 1} \left( \sum^M_{i = 1} (y_{j, i} - \mu)^2 + 2 \sum^M_{i > k}(y_{j, i} - \mu)(y_{j, k} - \mu) \right) \\ &= \frac{1}{N - 1} \left( \sum^N_{j = 1} \sum^M_{i = 1} (y_{j, i} - \mu)^2 + 2\sum^N_{j = 1} \sum^M_{i > k}(y_{j, i} - \mu)(y_{j, k} - \mu) \right) \\ \end{aligned} $$ Now we can multiple the first term by $\frac{N M - 1}{N M - 1}$ to get $S^2$ and we can multiple the second term by $\frac{(M - 1)(NM - 1) S^2}{(M - 1)(NM - 1) S^2}$ to get $\rho$.

$$ \begin{aligned} S^2_c &= \frac{1}{N - 1} \left( (N M - 1) S^2 + (M - 1)(NM - 1) S^2 \rho \right) \\ &= \frac{N M - 1}{N - 1} S^2 \cdot \left( 1 + (M - 1) \rho \right) \\ \end{aligned} $$ Substituting this value for $S^2_c$ in expression from Proposition 1.1 gives us

$$ \begin{aligned} V_{Cluster}(\hat{\mu}) &= \frac{S_c^2}{n \cdot M^2} \\ &= \frac{(N M - 1)}{ M(N - 1)} \frac{S^2}{n \cdot M} \cdot \left( 1 + (M - 1) \rho \right)\\ \end{aligned} $$ When $N$ is large, the second term is approximately equal to one, and we have the familiar expression

$$ \begin{aligned} V_{Cluster}(\hat{\mu}) \approx \frac{S^2}{n \cdot M} \cdot \left( 1 + (M - 1) \rho \right)\\ \end{aligned} $$

in which case the difference in $V_{SRS}$ and $V_{cluster}$ is equal to the last term which is referred to as the design effect.

These results can easily be extended to the case where $m$ units are sampled from each cluster.

I do not show this, but when clusters are not all the same size, the variance of a clustered sample is even greater.

Conclusion

It's possible (but rare) for $\rho$ to be negative in which case the variance of a clustered sample is smaller than the variance of a simple random sample. Otherwise, using $V_{SRS}$ in the place of $V_{Cluster}$ will underestimate the variance.

Underestimating the variance for the sample mean (for both the treatment and control group) result in a t-test that over-rejects the null.

Your second question regards the clustered standard errors in OLS.

Using a OLS with clustered-standard errors gives a FPR equal to the alpha.

Here, you just need to satisfy yourself that the estimated variance is the variance in Proposition 1.2 or Correlation 1.0. See this question and answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.