# A/B test with unequal sample size

We are measuring traffic and conversion rates on an eCommerce website. Conversion rate is defined as the % of traffic (users) that purchased something on the website out of all traffic over a period of time.

For example: If the total traffic was 1M over one week, and 100K users purchased something, then the conversion rate was 10% for that week.

Now I'm introducing some new feature to the users and want to test its influence on conversion rate. I run a split test over 1 week, whereby the total traffic on the website is split into two groups:

Test group (90% of users): see the new feature. Control group (10% of users): do not see the new feature.

The 90-10 split is done randomly by a computer and the sample size is in the millions.

Now I measure conversion rate for both groups. Suppose that after a week I get these results:

Total Visitors: 1M

Test group: 900K (90% of users). Of them, purchased: 90K (10% conversion)

Control group: 100K (10% of users). Of them, purchased: 5K (5% conversion)

Am I correct to assume that since the conversion rate in the test group is twice as large as in the control group, that means that the new feature has improved conversion by 200%, even though the size of the groups is not the same?

• I don't understand you choice of split 9:1, why 9:1? Why not 1:1? Commented Oct 17, 2014 at 11:11
• there are technical reasons for this. Let's assume i have to do it this way. Commented Oct 17, 2014 at 11:54
• Sure, but you should repeat the test at least, say, 3 times. Commented Oct 17, 2014 at 12:14
• I know there's not much hope at this point, but I'd dearly love to see the answer to this question. Commented Apr 18, 2019 at 15:34
• @BalRog, please see below for my answer Commented Aug 25, 2022 at 1:26