0
$\begingroup$

I need to compare two targeting models in an experiment. We will apply the same treatment to everyone who is targeted.

The experimental set up is as follows:

  • The total population is split into three groups: A (10K), B (10K), C (4k)
  • We then rank the population of Group A according to algorithm A and apply a treatment to the top 5k people.
  • We rank the population of Group B according to algorithm B, and apply the same treatment to the top 5k.
  • In Group C we do no ranking and apply no treatment.

My question is:

  • How do we compare the success of each group?
  • Do we compare the total number 'cured' after receiving the treatment in Groups A vs Group B?
  • Do we compare the total 'cured' in A vs B (both compared to the control group C)?
$\endgroup$
  • 1
    $\begingroup$ Just to be clear: The lower 5k from groups A & B are untreated, correct? The actual "success" depends directly to your metric but unfortunately this is not elucidated in the question. I would also note that the top 5K from groups A & B are not random samples, there is a clear selection bias based on ranking the users prior to providing them with treatment. $\endgroup$ – usεr11852 Jun 29 at 23:49
  • $\begingroup$ Yes, the lower 5k are untreated. The metric is either total 'cured' amongst the treated or total 'cured' amongst the whole group. We are not sure which to use, and why. $\endgroup$ – Tom Kealy Jul 1 at 9:03
  • $\begingroup$ Why would we use the latter? If anything 50% have not been treated so we just add noise to our metric. As mentioned though, there is a strong selection bias here because of the ranking step. $\endgroup$ – usεr11852 Jul 1 at 9:06
  • $\begingroup$ Why would we use the latter? There are those amongst the 5k who don't receive the 'treatment' who will go on to be 'cured' (the application isn't medical). I have been told that we have to include the possibility of them in our analysis. $\endgroup$ – Tom Kealy Jul 1 at 9:23
  • $\begingroup$ 'As mentioned though, there is a strong selection bias here because of the ranking step.' That's exactly right, and I don't understand how to compare these groups given the different selection biases in the two different ranking steps. $\endgroup$ – Tom Kealy Jul 1 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.