3
$\begingroup$

I am working on a consulting project where my client wants to perform AB testing in order to improve their service which involves outbound communication to clients. Their clients are (structurally) stratified into groups and due to technological limitations the client is unable to randomize the treatments within the groups.

So, at first I thought that a good way to approximate a fully randomized experiment is to choose pairs of groups that are `similar' and then assign the treatment and control within the pairs at random. Recently, the client sent me a list of a few dozen groups with a handful of descriptive statistics for each and asked how to group them. Since we don't have any outcome data, I am tempted to tell them to pair the groups based on their own judgment, but then I started doubting my recommended course of action a little bit. So, a few questions:

  • Do you think this method of pairing may induce bias in some way that I am not anticipating?
  • Do you think it might be better to just allocate the treatment and control groups completely at random? (Will increase variance but might reduce bias?)
  • Is there any possible benefit for using a clustering algorithm to decide on how to pair the groups rather than just ask the client to do it based on their expertise?

A clarification: We have different groups of clients, lets say groups $A$, $B$, $C$ and $D$. Each group may have a large number of clients associated with them, usually in the thousands. It is possible to assign a treatment or control at random at the group level, meaning that all clients in group $A$ and $B$ will get the treatment and groups $C$ and $D$ will be controls, but it is not possible to assign the treatment to just half the clients in group A (and assign the control to the rest).

Improve this question
$\endgroup$
4
  • $\begingroup$ You said at the top that you can't randomize within groups, but in bullet point 2 suggest that you can randomize at least something. Back up and explain what can and can't be randomized (in particular: what "technological limitations" are preventing randomization?). Random assignment is much more valuable than being able to pair treatment and control units. $\endgroup$
    – Kodiologist
    Aug 16, 2017 at 18:53
  • $\begingroup$ Thank you for the comment. I clarified in the question that we can assign treatment and control to entire groups of clients, but not randomize within groups. I can't expand on the technical limitations, but they have to do with the software used rather than with the research question. I recognize that randomization is more important than being able to pair treatment and control groups. However, the question is concerned with whether the pairing procedure, the purpose of which is to reduce variance, is likely to induce bias or not. $\endgroup$
    – user3903581
    Aug 16, 2017 at 20:32
  • $\begingroup$ "I can't expand on the technical limitations" — If you can't explain your question, I can't answer it. Sorry. $\endgroup$
    – Kodiologist
    Aug 16, 2017 at 20:48
  • 1
    $\begingroup$ @Kodiologist, what OP means is that the unit of analysis is the group. So, given a bunch of groups with known characteristics, what is the best way to randomize to reduce bias while maintaining variance? $\endgroup$
    – Noah
    Aug 23, 2017 at 20:47

1 Answer 1

Reset to default
1
$\begingroup$

There are some creative ways to get around this. In the clinical trail literature, this is sometimes referred to as a "cluster randomized design". The motivating example is usually studies involving nursing homes. Since you can't randomize individuals within nursing homes to treatment or control then the other option is to randomize everyone within the nursing home to either treatment or control.

I can't speak to the complexity of analysis of these designs. My intuition says that a hierarchical regression model could likely deal with any within-cluster correlation, but if you were to summarize the clusters in some way then the clusters could be considered independent.

In any case, the point is that this problem has been examined in the medical literature, and I bet if you were to adopt such a perspective then you could find a solution there.

Improve this answer
$\endgroup$
1
  • $\begingroup$ One could just cluster the standard errors in the outcome regression by each randomization unit (say nursing home). This allows the correlation of the error terms for residents within each home, but not across homes. $\endgroup$
    – dimitriy
    Mar 5, 2020 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.