# Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country).

The people from the segment are divided into 2 groups. Part of them see a banner and the other don't (control group). Usually the control group is 30% of visitors. The test begin after the banner is shown to all users in the segment for a while so the data of exposed people extends much longer than the data of the control group.

So at a given time we have for example X people exposed and Xb of them converted; likewise for Y people who were not exposed, Yb of them were converted. Y and Yb are much smaller than X and Xb. The conversion Rate for X is Xb/X and for Y is Yb/Y.

My first question is how to determine statistical validity.

We used Chi-square test to do it, and got results similar to those from this on-line calculator (implemented similar to table1 here). However, sometimes it looks like the number of purchases is extremely small yet the chi-square test says its valid (>95%).

Here is a real life example:

X=189 Xb=1 Y=93 Yb=3
Conversion X= 0.5% Conversion Y=3.2%
Statistical confidence (chi-square based) 92.8%


Altough the confidence is below 95%, it seems too close for determining its confidence while there was only 1 conversion for X.

My second question is then: Should there also be a requirement for a minimal number of conversions for the confidence to be valid? If so, how do we calculate it?

Thanks!

• I have made some edits to your question. Please, could you check that I didn't alter its original meaning?
– chl
Mar 8, 2011 at 11:34