1
$\begingroup$

Suppose I run AB testing for a website that does subscription business. The company offers a free trial for 7 days before automatically enrolling users to subscriptions and charging them unless they cancel before the trial ends.

The treatment is trial reminder message at the checkout page and a push notification before trial ends. Users arrive at the checkout page will enter the experiment (only test group will see the trial reminder message on checkout page and get notification after), but they will only start trial after they finished checkout by entering their credit card.

If we want to look at the impact on cancellation rate and average charge per user, do I use all users in the experiment as the denominator or only those who started the trial for both cancellation rate and average charge per user?

I feel we should use everyone in the experiment as denominator, but only those who started trial can cancel, hence using those who started trial can give us better measure of cancellation rate. However, if users starting trial at different rate between test and control group, I would have biased result for cancellation rate if I only include those who started trial?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Is there an assumption that the trail reminder may affect conversion rate (i.e. the rate of users actually checking out by entering their credit card)? $\endgroup$
    – Demetri Pananos
    Commented May 30, 2023 at 20:31
  • $\begingroup$ it's possible, that's one of the test hypothesis. $\endgroup$
    – ggb5317
    Commented May 30, 2023 at 21:14
  • $\begingroup$ OK, it seems you have multiple hypotheses about these interventions. My recommendation is to determine what is of principle importance. Do you a) Care about the isolated effect of each of these, or b) care about the total effect of the two. If the former, it would be easiest to run a test with 4 groups (control + each combination of treatments 1 and 2). If the latter, then just run treatment and control as I've described. The structure of the treatment(s) seems to make inference about isolated effect from the two group design difficult. $\endgroup$
    – Demetri Pananos
    Commented May 30, 2023 at 22:03

1 Answer 1

Reset to default
2
$\begingroup$

This is a very good question. Let's first outline a few assumptions and draw a DAG.

  • First, it seems like users are randomized after arriving at the checkout page. This means the "treatment" is applied prior to the checking out.
  • A reminder of a trial could possibly effect conversion per randomized user. Ostensibly, you are reminded that you will be paying in the future which may change the probability you convert.
  • There is an additional treatment applied conditional on having checked out.

With these in hand, a reasonable dag might be

enter image description here

OP correctly notes that conditioning on those users who enter their credit card (Enter CC in the dag) results in a bias. If I recall correctly, this would be selection bias (because there are unmeasured confounders (U in the dag) which might be associated with entering the credit card and conversion. One of these might be motivation. Motivated users are going to enter their credit card and covert because they really want to, which can bias estimates.

However, the resulting estimate from simply analyzing the experiment using an indicator for having seen the reminder (Treatment #1 in the dag) is not the direct effect of the reminder since the push notification (Treatment #2) is on the causal path to conversion.

There are a two options then:

  • Make the reminder the only treatment. Those users arriving on the checkout page enter the experiment and the denominator is the count of said users.
  • Make the push notification the only treatment. Those users entering the free trial enter the experiment and the denominator is the count of said users.

An additional option might exist so that some sort of causal effect for treatment #1 and #2 can be identified, but I think this would rely on mediation analysis which I am not convinced is as reliable as some may say. I'm about to leave for a little bit, but I intend to come back and flesh out my answer some more.

Lastly, this answer relies on the assumption that treatment #1 will affect credit card entry. The extent to which this is a reasonable assumption relies on knowledge only OP has.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This is great. The Dag is very helpful in understanding the treatment pathways. I agree it's cleanest to separate out the treatment effects, but in reality, these 2 should be bundled as a reminder copy on the checkout page does indicate user will get a reminder. Hence reminder copy + notification reminder is a bundled experiment. If we don't see a significant difference in credit card conversion rate, then can we safely define the denominator of the cancellation rate as those who entered credit card info? $\endgroup$
    – ggb5317
    Commented May 30, 2023 at 21:08
  • $\begingroup$ @ggb5317 If all you care about is the direct effect of both, rather than the effect of each treatment, then I think you're OK and you should use all randomized users (not just the ones who enter the free trial) $\endgroup$
    – Demetri Pananos
    Commented May 30, 2023 at 21:09
  • $\begingroup$ Agreed. But also wonder if it's ok to use those who entered CC as denominator if there is no significant changes in CC entry between the group. Reason being if we refine denominator as those who started trial, we will be able to detect larger effect, otherwise effect will be diluted as majority users don't start trial, and denominator with everyone will wash out the effect. $\endgroup$
    – ggb5317
    Commented May 30, 2023 at 21:17
  • $\begingroup$ @ggb5317 DAG would seem to imply there would be a bias. The size and direction is tough to say. Point taken on power and dilution. If that is your primary concern, why not just test treatment 2? $\endgroup$
    – Demetri Pananos
    Commented May 30, 2023 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.