3
$\begingroup$

I looked answers for this question but couldn't find anything clear, everyone says something different.

I wonder what is the most appropriate method for testing the statistical significance of our A/B testing results. Currently, we are using Chi-Sqr but I am not sure if it is good or not for us.

For example:

version A: 90,000 visitors, 50,000 purchases
version B: 45,000 visitors, 25,700 purchases

I want to test if there is a significant difference between the conversion rates of two versions.

Improve this question
$\endgroup$
9
  • $\begingroup$ More information is needed. Is the design pre-post? Is the response categorical? Are there 0 cell counts or otherwise small sample sizes? If the answers are No/Yes/No, then a (Pearson) "Chi-Square test" (of independence) is reasonable. $\endgroup$
    – AdamO
    Commented Jul 19, 2019 at 15:35
  • $\begingroup$ @AdamO I added an example to first post $\endgroup$
    – Erdal Dogan
    Commented Jul 19, 2019 at 16:03
  • $\begingroup$ Are you interesting in purchases (50000 vs 25700) or rate of purchases (50/90 vs 25.7/45)$? Generally, we are interesting on rate, but in your situation maybe you just focus on purchase. $\endgroup$
    – user158565
    Commented Jul 19, 2019 at 18:18
  • $\begingroup$ I focus on rates $\endgroup$
    – Erdal Dogan
    Commented Jul 19, 2019 at 19:14
  • $\begingroup$ What is called "conversion rate"? $\endgroup$
    – user158565
    Commented Jul 21, 2019 at 13:46

1 Answer 1

Reset to default
2
$\begingroup$

Here is a test of two proportions in Minitab. It uses a normal approximation which should be accurate for such large numbers of counts. Because the square of a standard normal distribution is chi-squared with one degree of freedom, a chi-squared test on a $2 \times 2$ table would be essentially the same. There are minor differences in how (or whether) various programs implement 'continuity correction'. 

Test and CI for Two Proportions 

Sample      X      N  Sample p
1       50000  90000  0.555556
2       25700  45000  0.571111

Difference = p (1) - p (2)
Estimate for difference:  -0.0155556
95% CI for difference:  (-0.0211635, -0.00994764)
Test for difference = 0 (vs ≠ 0):  
    Z = -5.44  P-Value = 0.000

The P-value is so small that it is hard to imagine a valid test would not find a significant difference between A and B proportions. [Minitab shows P-values to three places, so output 0.000 indicates a P-value below $0.0005.]$

It would have been easier to know what is puzzling you if you had shows differences from various tests. Your data seem to be severely rounded; you should use actual counts in such an analysis.

Note: If all four counts were divided by 100, then proportions would be the same, but they would not be significantly different. Sample size matters.

Sample    X    N  Sample p
1       500  900  0.555556
2       257  450  0.571111
...
Test for difference = 0 (vs ≠ 0):  
    Z = -0.54  P-Value = 0.587
Improve this answer
$\endgroup$
3
  • $\begingroup$ Instead of including the total numbers of visitors in the table, I used conversions and non-conversions(total_visits - conversion), since the visitors are directed the different versions by software and not equally distributed always. What I did was to calculate the expected conversions for each version ((total_conversions / total_visitors) * total_visits_on_version_x), then compared two 2x2 matrixes with an excel function (CHITEST). We have very close p-values though. I am not an expert on statistics, excuse my mistakes. $\endgroup$
    – Erdal Dogan
    Commented Jul 19, 2019 at 17:21
  • $\begingroup$ Results should be OK if total visitor counts and conversion rates are both accurate. But if the claimed numbers of total visitors were due to bragging rather than counting, you could be in trouble. // Have to be careful these days: politicians claim 'millions and millions' when they actually talked to half a dozen supporters. $\endgroup$
    – BruceET
    Commented Jul 19, 2019 at 17:30
  • $\begingroup$ No aggregated claims, no assumptions, no nothing :) They are precise numbers of visitors of a webpage. Thank you for your help. $\endgroup$
    – Erdal Dogan
    Commented Jul 19, 2019 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.