1
$\begingroup$

This is more of an experimental design question, but the implications are largely statistical, so I'd appreciate some help.

I'm running an experiment with traditional experimental and control groups, and we're expecting ~120 participants. The challenge is that it's an educational intervention, and our participants come from ~12 classrooms of varying sizes. Students in a given classroom need to have the same condition to ensure a consistent experience - that's unfortunately not negotiable.

Question 1: Given my 10 classrooms of varying sizes, how should I go about assigning them conditions?

Let's say the sizes are: 6, 15, 12, 17, 3, 4, 12, 14, 13, 10, 5, 5, 7, 5.

I could randomly assign classes to a condition, but then I risk very different sizes for the two conditions. I've thought of pairing up the classrooms by size (2 largest, second 2 largest, etc.) and then randomly splitting each pair. Does that raise any red flags?

Is there a better method that preserves as much randomness as possible, while maintaining fairly even condition sizes?

Question 2: Given students' condition X, classroom C and an outcome variable Y, how should I test for differences between my conditions?

I was planning on using an ANCOVA to control for C. Are there better methods? I'll likely need a nonparametric test as well (for ordinal or non-normal data). Any advice on how to do that?

Edit: I see that ANCOVA is inappropriate for two reasons here. 1) The variable I'd like to control for C is not continuous. 2) According to this answer it is also not appropriate when the covariate is related to experimental group, as in my case. Further, this paper suggests that "controlling" for non-random assignment is not possible.

Is there any way then to compensate for the limitations of my experimental design in analysis? Or will it be impossible to show that differences in Y are due to the intervention (X) as opposed to classroom (C).

Thanks for any input!

$\endgroup$
1
$\begingroup$

Question 1: I think the easiest way to handle this is to include classroom size as a covariate in your analysis. That should adjust for any differences in outcome due to classroom size. That way you can randomize at the classroom level in the normal fashion.

If this remains a concern, you could consider case-control methods, which also use non-random assignment. The weakness in this approach is the reduction in the causal strength of your claims and the concern about finding all the covariates of interest and either controlling for them or using a matching method, such as propensity score matching.

In your setup, propensity score matching seems like an acceptable strategy. Here, you'd match students based on characteristics (such as the classroom they are in, age, gender, SAT score), the look at the paired changes from the matched students in each of the cohorts. You'd still have a treatment effect using individual student data, but the strength of that claim would depend on the strength of the matching. This is a clear weakness compared to randomization, even at the classroom level.

Question 2: This analysis seems ideally suited for a mixed model design. You have sufficient classrooms to include it as a random effect, and that would adjust for differing baselines and/or slopes in your outcome measure. It would also allow you to easily include classroom size as a covariate. This paper by Laird and Ware discusses the advantages of a mixed model with respect to unbalanced designs. A mixed model with random classroom assignment eliminates the need for propensity score matching.

$\endgroup$
  • $\begingroup$ Thanks for the ideas. For Question 1, my concern about random assignment of classrooms to conditions is less about the class size covariate, and more about the possibility of large differences in condition sizes. Some simple simulation with the example sizes I gave you suggests an average difference of 30 participants (e.g. 50 vs 80). For comparison, random assignment per individual would have an average difference of 8. Isn't that a concern? $\endgroup$ – thomas88wp Jun 1 '17 at 13:59
  • $\begingroup$ If you ultimately use a mixed model design (instead of the ANCOVA, as you've already indicated you'd remove), then unbalanced designs are not a concern. It's only if you get very extreme imbalances that you should worry. $\endgroup$ – Ashe Jun 1 '17 at 14:03
  • 1
    $\begingroup$ The Laird and Ware paper in this answer talks a bit about the ease of handling unbalanced designs in a two-staged mixed model. Other helpful comments there about mixed models in that answer. stats.stackexchange.com/questions/764/… $\endgroup$ – Ashe Jun 1 '17 at 14:15
  • $\begingroup$ Thanks, for the resources - these are helpful! Let's say I'm still not 100% confident I'll be able to make a mixed model will work for my data, and I'd like to balance group sizes for peace of mind. Other than it being unusual, what would the statistical argument be against using a more balanced method, like the paired random splitting suggested in my question? $\endgroup$ – thomas88wp Jun 1 '17 at 14:44
  • $\begingroup$ The biggest advantage of randomization is that you can make stronger causal claims. By fully randomizing at the classroom level, you can say your condition increase or decreases your outcome when adjusting for covariates (classroom identity via random effect, classroom size, C, etc). If you don't randomize, then the effect you see could be due to some hidden criteria in your assignment method. You could consider adapting case-control methods, which have a similar lack of randomization element to the analysis. en.wikipedia.org/wiki/Case-control_study I edited the answer. $\endgroup$ – Ashe Jun 1 '17 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.