We are measuring traffic and conversion rates on an eCommerce website. Conversion rate is defined as the % of traffic (users) that purchased something on the website out of all traffic over a period of time.
For example: If the total traffic was 1M over one week, and 100K users purchased something, then the conversion rate was 10% for that week.
Now I'm introducing some new feature to the users and want to test its influence on conversion rate. I run a split test over 1 week, whereby the total traffic on the website is split into two groups:
Test group (90% of users): see the new feature. Control group (10% of users): do not see the new feature.
The 90-10 split is done randomly by a computer and the sample size is in the millions.
Now I measure conversion rate for both groups. Suppose that after a week I get these results:
Total Visitors: 1M
Test group: 900K (90% of users). Of them, purchased: 90K (10% conversion)
Control group: 100K (10% of users). Of them, purchased: 5K (5% conversion)
Am I correct to assume that since the conversion rate in the test group is twice as large as in the control group, that means that the new feature has improved conversion by 200%, even though the size of the groups is not the same?