Looking for the best way to set up an AB test.
Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features.
Each user has a 'propensity to upgrade' score that is calculated daily by a machine learning model. For simplicity, assume that this gives a score of high/medium/low (and this can change on a daily basis for each user).
We want to test 3x treatments:
- high propensity: when a user logs in, show a lightbox where you can book a meeting with a customer success person to demo new features in the product
- medium propensity: when a user logs in, show a lightbox where you can watch a video that showcases new features in the product
The success measure is if they upgraded within X days. The logic is that we are happy to have the high propensity users to connect to a human being, but not medium propensity users (given the higher cost). Low propensity users are shown nothing as we don't want to annoy them with upgrade messages.
How would you set up an AB test given that the propensity score can change daily, making the user eligible for multiple treatments? Would you just enrol a given user based on their first score (so today, they may have a medium propensity, but tomorrow they have a high propensity, however they're only ever shown then treatment for medium propensity?)