My customer performs targeted marketing campaigns on subscribers. For evaluation purposes he splits target audience into target and control groups (TG, CG). He says he does this splitting randomly, but actually he does it in a special way.
He makes dozens of iterations of true random splits, and in each iteration he compares pre-campaign data averages in CG and TG candidates in a split (like, past-month revenues, lifetimes, etc.). Finally, he chooses a split where pre-campaign differences between CG-TG candidates are minimal. He motivates this with a necessety of future comparison of CG and TG in post-campaign period, and this way he ensures maximum similarity of the groups before the start of the campaign.
I argue that random split is random by design, and there is no point to resplit several times to find out the 'best' one, as any difference (or absence of the difference in resulting TG and CG before the campaign) is random. He argues that as we finally settling groups, their pre-campaign data is not anymore random, and as such, we'd better settle them in a way where pre-campaigns differences are minimal.
As groups are large (usually, many thousands of subscribers), this resplitting approach is hardly influence pre-campaign comparability of the groups (it actually just make extra work load on database). And I assume the evaluation scheme remains statistically sound.
But if TG and CG would be smaller (and another random reassignment of the groups can make pre-campaign data indeed more or less balanced by KPI of interest), whose point of view would be correct? I mean, correct from the pure statistical and from practical (campaign evaluation for stake holders) points, if they somehow differ.