We did an A/B test where A is control and B is the same website with one feature variant that is available on all pages on the website. The feature variant is live chat. We wanted to see if live chat helps people convert or if it hinders.
In total A and B have similar conversion rates. Only 1% of users in group B used live chat. However, the those who did had a conversion rate 10 times higher.
One possibility could be that A and B have similar conversion rates because the percentage of users who use live-chat is so low. Another possibility is that the users who used live-chat were more engaged and so more likely to convert anyway.
Would it make sense to bootstrap the distribution of conversion rates for the population who used live chat and see how that distribution compared to the overall distribution? Or is there another way to tease out causality?