Is it worth redoing random split several times in order to draw the 'best' control group?

Question

My customer performs targeted marketing campaigns on subscribers. For evaluation purposes he splits target audience into target and control groups (TG, CG). He says he does this splitting randomly, but actually he does it in a special way.

He makes dozens of iterations of true random splits, and in each iteration he compares pre-campaign data averages in CG and TG candidates in a split (like, past-month revenues, lifetimes, etc.). Finally, he chooses a split where pre-campaign differences between CG-TG candidates are minimal. He motivates this with a necessety of future comparison of CG and TG in post-campaign period, and this way he ensures maximum similarity of the groups before the start of the campaign.

I argue that random split is random by design, and there is no point to resplit several times to find out the 'best' one, as any difference (or absence of the difference in resulting TG and CG before the campaign) is random. He argues that as we finally settling groups, their pre-campaign data is not anymore random, and as such, we'd better settle them in a way where pre-campaigns differences are minimal.

As groups are large (usually, many thousands of subscribers), this resplitting approach is hardly influence pre-campaign comparability of the groups (it actually just make extra work load on database). And I assume the evaluation scheme remains statistically sound.

But if TG and CG would be smaller (and another random reassignment of the groups can make pre-campaign data indeed more or less balanced by KPI of interest), whose point of view would be correct? I mean, correct from the pure statistical and from practical (campaign evaluation for stake holders) points, if they somehow differ.

Answer 1

I would say that your customer is right.

For my rationale, let's discuss why we do random assignment to groups in the first place. In an ideal world, each individual in the TG would correspond to exactly one individual in the CG, who are identical in all relevant attributes - both those we know about, and those we do not know about. If we have such a matching design, we know that differences in the outcome are due to the intervention.

Unfortunately, there are two problems with such a perfect matching. The first is that even matching on the known attributes is usually not possible, simply because there are too many possible combinations of attribute values. The second is that we usually do not know every relevant attribute, or are unable to measure them. This is why we do random group assigment, hoping that groups then do not differ greatly on the known attributes, and that the unknown attributes are similarly averaged out.

If we can assume that your customer's process does not induce systematic differences on the unknown attributes (and this assumption would IMO usually make sense), then we are left with the fact that it makes the groups more similar on the known attributes. Which is a step towards better matching, i.e., the ideal world.

So there is no downside, but there is upside. So your customer is right in his process.

Answer 2

It sounds like your customer wants a random split while controlling variance along specific metrics. Your customer's procedure is not wrong, but it is computationally expensive and difficult to reproduce (e.g. you need to store the seed of the best random assignment).

An alternative would be to perform a stratified random sample. The metrics of interest appear to be continuous, so this could be achieved by binning the candidates and randomly assigning within each bin. For example, if you require an 10/90 TG/CG split while controlling variance in terms of past-month revenues and lifetime, you could

1. Partition candidates into 25 bins where each bin contains candidates in the $i^{\mathit{th}}$ quintile of past-month revenue and the $j^{\mathit{th}}$ quintile of lifetime.
2. Within each bin, hash the candidate identifier concatenated with the experiment identifier to get a stable pseudorandom integer.
3. Select the 10% of candidates with the smallest hash values for TG.

This procedure guarantees that 10% of each bin are TG. You will probably want to tune the bin boundaries and the number of bins so that you do not end up with some bins with very few candidates (e.g. quintiles may not be the right criterion for binning).