Comparisons of targeting models

I need to compare two targeting models in an experiment. We will apply the same treatment to everyone who is targeted.

The experimental set up is as follows:

• The total population is split into three groups: A (10K), B (10K), C (4k)
• We then rank the population of Group A according to algorithm A and apply a treatment to the top 5k people.
• We rank the population of Group B according to algorithm B, and apply the same treatment to the top 5k.
• In Group C we do no ranking and apply no treatment.

My question is:

• How do we compare the success of each group?
• Do we compare the total number 'cured' after receiving the treatment in Groups A vs Group B?
• Do we compare the total 'cured' in A vs B (both compared to the control group C)?
• Just to be clear: The lower 5k from groups A & B are untreated, correct? The actual "success" depends directly to your metric but unfortunately this is not elucidated in the question. I would also note that the top 5K from groups A & B are not random samples, there is a clear selection bias based on ranking the users prior to providing them with treatment. – usεr11852 Jun 29 at 23:49
• Yes, the lower 5k are untreated. The metric is either total 'cured' amongst the treated or total 'cured' amongst the whole group. We are not sure which to use, and why. – Tom Kealy Jul 1 at 9:03
• Why would we use the latter? If anything 50% have not been treated so we just add noise to our metric. As mentioned though, there is a strong selection bias here because of the ranking step. – usεr11852 Jul 1 at 9:06
• Why would we use the latter? There are those amongst the 5k who don't receive the 'treatment' who will go on to be 'cured' (the application isn't medical). I have been told that we have to include the possibility of them in our analysis. – Tom Kealy Jul 1 at 9:23
• 'As mentioned though, there is a strong selection bias here because of the ranking step.' That's exactly right, and I don't understand how to compare these groups given the different selection biases in the two different ranking steps. – Tom Kealy Jul 1 at 9:24