0
$\begingroup$

We conducted A/B/C test in our website and now i want to check the results. But the problem i faced is that there are 2 test groups and 1 control, and not 1 test and 1 control groups as usual. When there is 1 control group and 1 test group I use binomial test and it's ok. If I had quantitative data I'd use ANOVA but my data is categorical. And since there is more that 1 test group I don't know which statistical criteria I should use to get correct results and understand which group is better for us.

The data structure's like:

group      |  number of visitors |  number of converted to order
-----------+---------------------+------------------------------
Control(A) |              159892 |                          2560
Test (B)   |              160201 |                          2754
Test (C)   |              159898 |                          2690
$\endgroup$
1
$\begingroup$

its a bit uncelar what you are asing for but maybe you could use a ordinal regression. This is if you want to predict.

https://en.wikipedia.org/wiki/Ordinal_regression

Or a Chi square. This is if you want to compare proportions of the groups. https://en.wikipedia.org/wiki/Chi-squared_test

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! Yes, i want to compare proportions ( B vs A and C vs A) and check whether there is a significant difference. How do you think should i use, for example, Tukey's method after? $\endgroup$ – Armen Jan Mar 17 at 11:51
1
$\begingroup$

I agree with Christian's answer, but I'd like to add some explanation.

I assume you want to find out if one version of the website is better than other, in terms of converting more visitors into customers. So you are actually comparing the fractions of customers among the visitors between the three groups.

Chi-squared test on a $2 \times 3$ table will give you an answer whether they differ significantly, but not which one of them is the best. If they differ, you should take the one with the highest fraction.

Edit:

For your concrete data, B is the best, but not too strongly:

n = matrix(c(
  159892, 2560,
  160201, 2754,
  159898, 2690), nrow=3, byrow=T
)
chisq.test(n)

results in:

    Pearson's Chi-squared test

data:  n
X-squared = 6.8843, df = 2, p-value = 0.032

The fractions are:

0.01601081
0.01719090
0.01682322
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Igor, thank you for your answer! I'll duplicate my question for Christian answer - should i use some post correction of p-value like Tukey's or Bonferroni's ? $\endgroup$ – Armen Jan Mar 17 at 12:34
  • $\begingroup$ No, I wouldn't do that. The chi-squared test is only one test and it already gives you the answer regarding the significance. $\endgroup$ – Igor F. Mar 17 at 12:55
  • $\begingroup$ And one more question - if i get p-value < 0.05, how can i be sure that the test catches the difference between A and B groups and not between B and C, for example (in cases where conversion rate of C group is the smallest)? $\endgroup$ – Armen Jan Mar 17 at 15:02
  • $\begingroup$ You can repeat the chi-squared test on $2 \times 2$ tables, for all pairs. In principle, if you'd have many such pairs, you could use the Tukey procedure, but in your case I don't think it's necessary. If both B and C have a better conversion rate than A, and one of them is significantly better then A, do you really want to know whether they significantly differ between themselves? Now, for your hypothetical case, where C might be actually worse than A, just make a post-hoc test between A and B, without correction. $\endgroup$ – Igor F. Mar 18 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.