6
$\begingroup$

We are measuring traffic and conversion rates on an eCommerce website. Conversion rate is defined as the % of traffic (users) that purchased something on the website out of all traffic over a period of time.

For example: If the total traffic was 1M over one week, and 100K users purchased something, then the conversion rate was 10% for that week.

Now I'm introducing some new feature to the users and want to test its influence on conversion rate. I run a split test over 1 week, whereby the total traffic on the website is split into two groups:

Test group (90% of users): see the new feature. Control group (10% of users): do not see the new feature.

The 90-10 split is done randomly by a computer and the sample size is in the millions.

Now I measure conversion rate for both groups. Suppose that after a week I get these results:

Total Visitors: 1M

Test group: 900K (90% of users). Of them, purchased: 90K (10% conversion)

Control group: 100K (10% of users). Of them, purchased: 5K (5% conversion)

Am I correct to assume that since the conversion rate in the test group is twice as large as in the control group, that means that the new feature has improved conversion by 200%, even though the size of the groups is not the same?

$\endgroup$
6
  • $\begingroup$ I don't understand you choice of split 9:1, why 9:1? Why not 1:1? $\endgroup$
    – eXpander
    Oct 17, 2014 at 11:11
  • $\begingroup$ there are technical reasons for this. Let's assume i have to do it this way. $\endgroup$
    – staccato
    Oct 17, 2014 at 11:54
  • $\begingroup$ Sure, but you should repeat the test at least, say, 3 times. $\endgroup$
    – eXpander
    Oct 17, 2014 at 12:14
  • $\begingroup$ I know there's not much hope at this point, but I'd dearly love to see the answer to this question. $\endgroup$ Apr 18, 2019 at 15:34
  • $\begingroup$ @BalRog, please see below for my answer $\endgroup$
    – wahid
    Aug 25, 2022 at 1:26

1 Answer 1

1
$\begingroup$

(This question is eight years old, but I'll take a shot to help folks in a similar situation)

The standard approach to validating differences in experiment group results is with a t-test. That tells you if the observed difference in means between the two groups is statistically meaningful (for a given significance level).

Practical significance is at least as important as statistical significance, if not more. In your example, depending on the return on investment from the resulting change, 200% is a practically important difference and is certainly worth a closer look (be it in the form of another experiment or just post-hoc analysis).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.