A/B test with unequal sample size

Question

We are measuring traffic and conversion rates on an eCommerce website. Conversion rate is defined as the % of traffic (users) that purchased something on the website out of all traffic over a period of time.

For example: If the total traffic was 1M over one week, and 100K users purchased something, then the conversion rate was 10% for that week.

Now I'm introducing some new feature to the users and want to test its influence on conversion rate. I run a split test over 1 week, whereby the total traffic on the website is split into two groups:

Test group (90% of users): see the new feature. Control group (10% of users): do not see the new feature.

The 90-10 split is done randomly by a computer and the sample size is in the millions.

Now I measure conversion rate for both groups. Suppose that after a week I get these results:

Total Visitors: 1M

Test group: 900K (90% of users). Of them, purchased: 90K (10% conversion)

Control group: 100K (10% of users). Of them, purchased: 5K (5% conversion)

Am I correct to assume that since the conversion rate in the test group is twice as large as in the control group, that means that the new feature has improved conversion by 200%, even though the size of the groups is not the same?

Answer 1

(This question is eight years old, but I'll take a shot to help folks in a similar situation)

The standard approach to validating differences in experiment group results is with a t-test. That tells you if the observed difference in means between the two groups is statistically meaningful (for a given significance level).

Practical significance is at least as important as statistical significance, if not more. In your example, depending on the return on investment from the resulting change, 200% is a practically important difference and is certainly worth a closer look (be it in the form of another experiment or just post-hoc analysis).