0
$\begingroup$

We did an A/B test where A is control and B is the same website with one feature variant that is available on all pages on the website. The feature variant is live chat. We wanted to see if live chat helps people convert or if it hinders.

In total A and B have similar conversion rates. Only 1% of users in group B used live chat. However, the those who did had a conversion rate 10 times higher.

One possibility could be that A and B have similar conversion rates because the percentage of users who use live-chat is so low. Another possibility is that the users who used live-chat were more engaged and so more likely to convert anyway.

Would it make sense to bootstrap the distribution of conversion rates for the population who used live chat and see how that distribution compared to the overall distribution? Or is there another way to tease out causality?

$\endgroup$
4
  • 1
    $\begingroup$ What exactly differed in the conditions of the A group versus the B group? Bootstrapping doesn't seem to have anything to do with the problem. $\endgroup$ – AdamO May 5 at 19:49
  • $\begingroup$ A is the control. B is the variant. The feature in B that is different is used only 1% of the time. Traditional hypothesis testing techniques find no significant difference. I was thinking bootstrapping could be used on the subpopulation and compare to the overall sample population. $\endgroup$ – Statbie May 5 at 20:24
  • $\begingroup$ That is not what I'm asking. What is the "variant"? Is this "feature" the variant? A short, maybe one paragraph, context is always helpful. $\endgroup$ – AdamO May 5 at 20:31
  • $\begingroup$ Thanks, I edited the question and hope that is clearer. $\endgroup$ – Statbie May 5 at 20:56
1
$\begingroup$

In a randomized controlled trial we often distinguish between the per-protocol versus the intent-to-treat effects of a drug. For instance, a highly effective anti-cancer drug may have low tolerability, so if you restrict analyses on patients who actually use the drug, you no longer have a randomized comparison, and the "compliers" in the active arm are generally a healthier or more resilient set.

Whether the actual use of the newly adopted chat feature in the B group led to a higher conversion rate is moreover concerned with the "per protocol" effect. But the fact that conversion was higher regardless of utilizing the feature may be part of an intent-to-treat story: the fact that the option was made available was reason enough to suggest the site was high enough quality to warrant a conversion. There would be no comparable "per protocol" analysis since there is no placebo or dummy condition in the "A" group by which to self-select the utilizers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.