We are measuring traffic and conversion rates on an eCommerce website. Conversion rate is defined as the % of traffic (users) that purchased something on the website out of all traffic over a period of time.

For example: If the total traffic was 1M over one week, and 100K users purchased something, then the conversion rate was 10% for that week.

Now I'm introducing some new feature to the users and want to test its influence on conversion rate. I run a split test over 1 week, whereby the total traffic on the website is split into two groups:

Test group (90% of users): see the new feature. Control group (10% of users): do not see the new feature.

The 90-10 split is done randomly by a computer and the sample size is in the millions.

Now I measure conversion rate for both groups. Suppose that after a week I get these results:

Total Visitors: 1M

Test group: 900K (90% of users). Of them, purchased: 90K (10% conversion)

Control group: 100K (10% of users). Of them, purchased: 5K (5% conversion)

Am I correct to assume that since the conversion rate in the test group is twice as large as in the control group, that means that the new feature has improved conversion by 200%, even though the size of the groups is not the same?

  • $\begingroup$ I don't understand you choice of split 9:1, why 9:1? Why not 1:1? $\endgroup$ – eXpander Oct 17 '14 at 11:11
  • $\begingroup$ there are technical reasons for this. Let's assume i have to do it this way. $\endgroup$ – staccato Oct 17 '14 at 11:54
  • $\begingroup$ Sure, but you should repeat the test at least, say, 3 times. $\endgroup$ – eXpander Oct 17 '14 at 12:14
  • $\begingroup$ I know there's not much hope at this point, but I'd dearly love to see the answer to this question. $\endgroup$ – BalRog Apr 18 '19 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.