We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country).
The people from the segment are divided into 2 groups. Part of them see a banner and the other don't (control group). Usually the control group is 30% of visitors. The test begin after the banner is shown to all users in the segment for a while so the data of exposed people extends much longer than the data of the control group.
So at a given time we have for example X people exposed and Xb of them converted; likewise for Y people who were not exposed, Yb of them were converted. Y and Yb are much smaller than X and Xb. The conversion Rate for X is Xb/X and for Y is Yb/Y.
My first question is how to determine statistical validity.
We used Chi-square test to do it, and got results similar to those from this on-line calculator (implemented similar to table1 here). However, sometimes it looks like the number of purchases is extremely small yet the chi-square test says its valid (>95%).
Here is a real life example:
X=189 Xb=1 Y=93 Yb=3
Conversion X= 0.5% Conversion Y=3.2%
Statistical confidence (chi-square based) 92.8%
Altough the confidence is below 95%, it seems too close for determining its confidence while there was only 1 conversion for X.
My second question is then: Should there also be a requirement for a minimal number of conversions for the confidence to be valid? If so, how do we calculate it?
Thanks!
