Is mutual exclusivity important for an A/B test for an audience selection method?

Question

Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are:

1. Take the entire population of 200 people and randomly split into two groups of 100.
2. From the first 100, randomly find 50 people. This is Group A.
3. From the second 100, you want to find 50 people that meet the business rules. However, only 40 people meet the business rules, so you have to find the last 10 using a random selection.
4. Send the same email to both groups and measure the response rates.

But, there are 10 people in Group A who you know satisfy the business rules.

Is it okay to include those 10 people in the Group B response rate instead of the 10 randomly selected? They have still been sent only one email, and it's the same email content. I can't see that mutual exclusivity is a requirement here, so it seems they can be included in both groups when measuring the response rate.

Answer 1

In my opinion, I would avoid this approach. If you start to move around people, e.g. discarding users that you don't like and re-sampling again, I would not consider this as random anymore. At the end, when you perform a random sample you want to study your sample and extract general conclusions about the respective population: if you modify your random sample, it might not be representative anymore. If some users in Group B do not meet the business rules, you must take this into consideration when comparing the performance of the two methods. Think, for example, that if the rules are too strict, and only a few users benefit from them, then the overall performance of your method won't be as good and you need to see this in your test.

If you want to analyze separately users that meet the business rule vs users that don't, you could have a Stratified Sampling approach (see here). But keep in mind that in the real case, your results will be affected by the share of users that actually meet the business rules.