# Statistical significance and winner of multivalent test

This should be a fairly straight forward, but I can't seem to figure out the answer. I'm doing some A/B(C/D/E...) testing on a website and measuring impressions and clicks. What method should I be using to determine the statistically significant winner?

This is sample data that I would have. There could be any number of tests, all being displayed in relatively-equal distribution and measuring clicks as a successful result. Such as:

Test #  |  Impressions  | Clicks
1     |      50       |   12
2     |      55       |   15
3     |      53       |   30
4     |      49       |   22


What algorithm should I be using to determine the winner in a statistically significant way?

If you are trying to relate number of clicks or number of impressions by group, I suggest a count regression model (Poisson or negative binomial, for instance) with "test #" as a categorical independent variable.

I'm not a statistician, but I've built a number of tools to evaluate statistical significance of A/B/n tests for inclusion in marketing tools. I'd love to have someone with more experience comment on my answer.

### Pearson's $\chi^2$ Test

One popular choice, and the one that produces the most consistent verifiable results is Pearson's Chi-Square Test. The math involved is straight forward and pretty easy to follow.

$$\chi^2=\sum_i\frac{(O_i-E_i)^2}{E_i}$$

Where $O_i$ is the observed value of clicks or no-clicks. $E_i$ is the observed values for clicks times the percentage of clicks, similar for no-clicks. In your example, Test 1 data is:

$$O_1c=12, O_1nc=38$$ $$E_1c=19.1, E_1nc=30.9$$

For online A/B/n testing you'll calculate these values for clicks & no actions in each group, so in your example you'll go through these steps a total of 8 times. Apply them to the equation above and you should find that in your example $\chi^2=15.6$.

Once you've calculated $\chi^2$ you use linear interpolation of a $\chi^2$ table to find the value of $p$.

$$p=y_0+(y_1-y_0)\frac{\chi^2-x_0}{x_1-x_0}$$

The degrees of freedom will always be 1 less than the number of observations. You had 8 observations (four sets of clicks & four sets of no-clicks), so in this case there are 7 degrees of freedom. Finally, for confidence interval, $CI = 1-p$. Carrying out your example, $p=0.03$, so Test 3 is the winner with a confidence of 97%.

### G-Test

Another cited choice is the G-test. This is closely related and should yield comparable results to the $\chi^2$ test.

$$G=2\sum_i O_i\centerdot ln(\frac{O_i}{E_i})$$

where $O_i$ & $E_i$ are calculated in the same way as above. Repeat this for all 8 of your attributes and you should end up with $G=15.7$. Use the same $\chi^2$ table as above and you'll find that $p=0.029$ and confidence is once again 97%.

### Z-Score

There are some articles online that recommend using the Z-score (AKA standard score). I disagree with this approach but include it for the sake of completeness.

$$z=\frac{p-p_c}{\sqrt{\frac{p(1-p)}{N}+\frac{p_c(1-p_c)}{N_c}}}$$

Where $p$ is the conversion rate (online this is referred to as CTR) of one of your elements, $N$ is the number of impressions for that element, $p_c$ and $N_c$ are the CTR & impressions for the control group. To calculate the Z-score for your winning group, we'll use:

$$p_c = 0.24, N_c = 50$$ $$p = 0.566, N = 53$$

Plugging these into the equation above, you'll get $Z=3.58$. As long as $Z>1.96$ you have a result that reaches 95% confidence. If you were using this as your criteria, you'd calculate the Z-score for each of your variants and take the highest.

The reason that I disagree with this approach is that there are situations for which I cannot verify the results via another method. If I use the data listed below:

               Impressions      Clicks
Control                50          12
Variation 1            55          23


I get a Z-score of 1.98, which by that test's criteria shows significance. However, if I try these same values in a $\chi^2$ calculator I find confidence far below 90%. I assume this is because the total number of impressions is low, which intuitively makes sense to me.

I'm not a statistician, so I can't do better than guess at the reason, but this leads me to trust Pearson's $\chi^2$ test more. My choice of the $\chi^2$ test over the G-test is arbitrary. I chose it first and have stuck with it, though I can say that I've also tried the G-test a few times and the results are always comparable.

I hope this helps. I also hope that someone with greater understanding can comment on my methods & choices.

• How did you calculate E(1c). Can you please elaborate? – Arpit Rai Oct 21 '16 at 12:41
• (Row total * Column total) / total n for the table. You can find a full explanation here. – Lenwood Oct 28 '16 at 5:52