I am working on a consulting project where my client wants to perform AB testing in order to improve their service which involves outbound communication to clients. Their clients are (structurally) stratified into groups and due to technological limitations the client is unable to randomize the treatments within the groups.
So, at first I thought that a good way to approximate a fully randomized experiment is to choose pairs of groups that are `similar' and then assign the treatment and control within the pairs at random. Recently, the client sent me a list of a few dozen groups with a handful of descriptive statistics for each and asked how to group them. Since we don't have any outcome data, I am tempted to tell them to pair the groups based on their own judgment, but then I started doubting my recommended course of action a little bit. So, a few questions:
- Do you think this method of pairing may induce bias in some way that I am not anticipating?
- Do you think it might be better to just allocate the treatment and control groups completely at random? (Will increase variance but might reduce bias?)
- Is there any possible benefit for using a clustering algorithm to decide on how to pair the groups rather than just ask the client to do it based on their expertise?
A clarification: We have different groups of clients, lets say groups $A$, $B$, $C$ and $D$. Each group may have a large number of clients associated with them, usually in the thousands. It is possible to assign a treatment or control at random at the group level, meaning that all clients in group $A$ and $B$ will get the treatment and groups $C$ and $D$ will be controls, but it is not possible to assign the treatment to just half the clients in group A (and assign the control to the rest).