We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country).

The people from the segment are divided into 2 groups. Part of them see a banner and the other don't (control group). Usually the control group is 30% of visitors. The test begin after the banner is shown to all users in the segment for a while so the data of exposed people extends much longer than the data of the control group.

So at a given time we have for example X people exposed and Xb of them converted; likewise for Y people who were not exposed, Yb of them were converted. Y and Yb are much smaller than X and Xb. The conversion Rate for X is Xb/X and for Y is Yb/Y.

My first question is how to determine statistical validity.

We used Chi-square test to do it, and got results similar to those from this on-line calculator (implemented similar to table1 here). However, sometimes it looks like the number of purchases is extremely small yet the chi-square test says its valid (>95%).

Here is a real life example:

X=189 Xb=1 Y=93 Yb=3
Conversion X= 0.5% Conversion Y=3.2% 
Statistical confidence (chi-square based) 92.8%

Altough the confidence is below 95%, it seems too close for determining its confidence while there was only 1 conversion for X.

My second question is then: Should there also be a requirement for a minimal number of conversions for the confidence to be valid? If so, how do we calculate it?


  • $\begingroup$ I have made some edits to your question. Please, could you check that I didn't alter its original meaning? $\endgroup$
    – chl
    Mar 8, 2011 at 11:34

1 Answer 1


I am not be able to answer your question completely, but are you looking for this: Power analysis - http://www.statmethods.net/stats/power.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.