AB Test, Experimental Design based on Machine Learning Output

Question

Looking for the best way to set up an AB test.

Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features.

Each user has a 'propensity to upgrade' score that is calculated daily by a machine learning model. For simplicity, assume that this gives a score of high/medium/low (and this can change on a daily basis for each user).

We want to test 3x treatments:

• high propensity: when a user logs in, show a lightbox where you can book a meeting with a customer success person to demo new features in the product
• medium propensity: when a user logs in, show a lightbox where you can watch a video that showcases new features in the product

The success measure is if they upgraded within X days. The logic is that we are happy to have the high propensity users to connect to a human being, but not medium propensity users (given the higher cost). Low propensity users are shown nothing as we don't want to annoy them with upgrade messages.

How would you set up an AB test given that the propensity score can change daily, making the user eligible for multiple treatments? Would you just enrol a given user based on their first score (so today, they may have a medium propensity, but tomorrow they have a high propensity, however they're only ever shown then treatment for medium propensity?)

Answer 1

Somewhat anticlimactically I would suggest not allowing users to be eligible for multiple treatments during the duration of the study. Your idea of enrolling them based on their first score is probably the best.

Reasons:

1. Changing treatments will most likely invalidate the ignorability assumption, i.e. that there is independence between our treatment status $Z$ and the potential outcomes we might have. Identification, Inference and Sensitivity Analysis for Causal Mediation Effects (2010) by Imai et al. give an in-depth discussion of implications of violating ignorability assumptions.
2. Changing treatments will make our users susceptible to priming effects from previous treatments, i.e. your previous treatment status $Z_{t=0}$ might interfere with your later response at $t=1$. This is a well-known effect, Yi (1990) The Effects of Contextual Priming in Print Advertisements is one of the first reference on the matter. I have seen some blog-posts flying around too (e.g. here) but research is more scare there.
3. Changing treatments, based on potentially unobserved confounders, will most likely invalidate our positivity assumptions for an individual user. (Positivity is violated if within a particular segment of our population (e.g. high-propensity users) nobody receives a particular treatment (e.g. "video treatment")) Diagnosing and responding to violations in the positivity assumption is some wordy for my liking but quite comprehensive if you want a more formal treatment.

And these above points are not even touching on a fourth potential issue where the ML model assigning propensities can create a feedback loop with the treatment status $Z$ itself. (There are no good solutions in that case, CV.SE has an interesting thread on the matter here: How to deal with cycles in causal inference?)

In short, users changing treatment is a minefield in terms of drawing valid insights from our A/B test. We will still have plenty to deal with (e.g. excluding people who could never be eligible for both treatments, time-varying effects, etc.) without the extra hassle of users changing treatment arms.