Consider a cluster-randomized experiment. There are 4,000 clusters and 2,000,000 observations. The dependent variable y is dichotomous, $Y \in \{0, 1\}$, and measured at the individual level. The independent variable x is also dichotomous, $X \in \{0, 1\}$, and randomized at the cluster level.

Modeling attempts.

I tried using a multilevel model using both lme4::glmer() and rstanarm::stan_glmer() with logistic link functions, using the formula y ~ x + (1 | id). The former doesn't converge and the latter did not advance even to 10% of the way through sampling, even on 8 cores and after hours. I tried a gee, but ran out of working memory—it told me a vector was too large after fitting the base glm().

I resolved to run a glm() with a logistic link function and the formula y ~ x. I then obtained a cluster-robust sandwich variance-covariance matrix from sandwich::vcovCL() and multiwayvcov::cluster.vcov() (they returned the same matrix).

Cluster-robust SEs.

These cluster-robust standard errors (SEs) were quite large. In one of the experimental conditions, the 95% CI on a proportion was a little more than 7ppts. I feel like this was too large for a dataset with 2mm observations and 4k clusters.

Possible problem.

My cluster sizes vary wildly in size. This histogram shows the size of clusters. As you can see, very many are less than 100, while some are upwards of 10,000 in size.

enter image description here

Here is a summary() of the counts by cluster:

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1.0    10.0    43.0   467.4   313.0 13089.0 


Could this skewed distribution of cluster sizes be responsible for the SEs that I find subjectively to be too large, given the amount of data I have? Could it be responsible for some of the optimization problems I run into with my mixed models? In general, is it an issue that 25% of clusters are 10 observations or less, while 25% are also 313 or more—going up to 13k?

  • 1
    $\begingroup$ Is there separation? Ie, are there clusters where one level of x has only 1's or only 0's for y? $\endgroup$ – gung - Reinstate Monica Jun 18 '19 at 18:22
  • 1
    $\begingroup$ Yes, the minimum cluster size was 1, so by definition there is separation there. Randomization was done at the cluster level, so each cluster has only one level of x. I took my data, grouped by cluster, and got the mean of the response variable. If the mean was exactly 1 or 0, I counted this as separation. Based on this, I see 452 clusters in my data where there are separation (out of 4k clusters). $\endgroup$ – Mark White Jun 18 '19 at 18:35
  • 1
    $\begingroup$ My hunch is that's your problem (at least w/ the frequentist model). I believe a Bayesian model w/ an 'appropriate' prior can get past that (but someone w/ more expertise should confirm). $\endgroup$ – gung - Reinstate Monica Jun 18 '19 at 18:44
  • $\begingroup$ Thanks. This makes sense intuitively: If there is no variance, there is no variance to explain. Could this explain why sandwich SEs are so large? I mean, you can't get more correlation within cluster than everyone having the same behavior. Also, do you have a reference on separation and why it causes issues? $\endgroup$ – Mark White Jun 18 '19 at 18:45
  • 1
    $\begingroup$ My thinking is slightly different. On the scale of the linear predictor, the random intercepts have to be normally distributed, w/ the data randomly dispersed around the underlying mean probability. But the logit of, say, $0$ is $-\infty$, which causes a variety of problems. I'm not aware of a reference, but there presumably is one. $\endgroup$ – gung - Reinstate Monica Jun 18 '19 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.