Assigning experimental conditions when subgroups must have the same condition

Question

This is more of an experimental design question, but the implications are largely statistical, so I'd appreciate some help.

I'm running an experiment with traditional experimental and control groups, and we're expecting ~120 participants. The challenge is that it's an educational intervention, and our participants come from ~12 classrooms of varying sizes. Students in a given classroom need to have the same condition to ensure a consistent experience - that's unfortunately not negotiable.

Question 1: Given my 10 classrooms of varying sizes, how should I go about assigning them conditions?

Let's say the sizes are: 6, 15, 12, 17, 3, 4, 12, 14, 13, 10, 5, 5, 7, 5.

I could randomly assign classes to a condition, but then I risk very different sizes for the two conditions. I've thought of pairing up the classrooms by size (2 largest, second 2 largest, etc.) and then randomly splitting each pair. Does that raise any red flags?

Is there a better method that preserves as much randomness as possible, while maintaining fairly even condition sizes?

Question 2: Given students' condition X, classroom C and an outcome variable Y, how should I test for differences between my conditions?

I was planning on using an ANCOVA to control for C. Are there better methods? I'll likely need a nonparametric test as well (for ordinal or non-normal data). Any advice on how to do that?

Edit: I see that ANCOVA is inappropriate for two reasons here. 1) The variable I'd like to control for C is not continuous. 2) According to this answer it is also not appropriate when the covariate is related to experimental group, as in my case. Further, this paper suggests that "controlling" for non-random assignment is not possible.

Is there any way then to compensate for the limitations of my experimental design in analysis? Or will it be impossible to show that differences in Y are due to the intervention (X) as opposed to classroom (C).

Thanks for any input!

Answer 1

Question 1: I think the easiest way to handle this is to include classroom size as a covariate in your analysis. That should adjust for any differences in outcome due to classroom size. That way you can randomize at the classroom level in the normal fashion.

If this remains a concern, you could consider case-control methods, which also use non-random assignment. The weakness in this approach is the reduction in the causal strength of your claims and the concern about finding all the covariates of interest and either controlling for them or using a matching method, such as propensity score matching.

In your setup, propensity score matching seems like an acceptable strategy. Here, you'd match students based on characteristics (such as the classroom they are in, age, gender, SAT score), the look at the paired changes from the matched students in each of the cohorts. You'd still have a treatment effect using individual student data, but the strength of that claim would depend on the strength of the matching. This is a clear weakness compared to randomization, even at the classroom level.

Question 2: This analysis seems ideally suited for a mixed model design. You have sufficient classrooms to include it as a random effect, and that would adjust for differing baselines and/or slopes in your outcome measure. It would also allow you to easily include classroom size as a covariate. This paper by Laird and Ware discusses the advantages of a mixed model with respect to unbalanced designs. A mixed model with random classroom assignment eliminates the need for propensity score matching.