5
$\begingroup$

This should be a fairly straight forward, but I can't seem to figure out the answer. I'm doing some A/B(C/D/E...) testing on a website and measuring impressions and clicks. What method should I be using to determine the statistically significant winner?

This is sample data that I would have. There could be any number of tests, all being displayed in relatively-equal distribution and measuring clicks as a successful result. Such as:

Test #  |  Impressions  | Clicks
  1     |      50       |   12
  2     |      55       |   15
  3     |      53       |   30
  4     |      49       |   22

What algorithm should I be using to determine the winner in a statistically significant way?

$\endgroup$

2 Answers 2

3
$\begingroup$

If you are trying to relate number of clicks or number of impressions by group, I suggest a count regression model (Poisson or negative binomial, for instance) with "test #" as a categorical independent variable.

$\endgroup$
3
$\begingroup$

I'm not a statistician, but I've built a number of tools to evaluate statistical significance of A/B/n tests for inclusion in marketing tools. I'd love to have someone with more experience comment on my answer.

Pearson's $\chi^2$ Test

One popular choice, and the one that produces the most consistent verifiable results is Pearson's Chi-Square Test. The math involved is straight forward and pretty easy to follow.

$$\chi^2=\sum_i\frac{(O_i-E_i)^2}{E_i}$$

Where $O_i$ is the observed value of clicks or no-clicks. $E_i$ is the observed values for clicks times the percentage of clicks, similar for no-clicks. In your example, Test 1 data is:

$$O_1c=12, O_1nc=38$$ $$E_1c=19.1, E_1nc=30.9$$

For online A/B/n testing you'll calculate these values for clicks & no actions in each group, so in your example you'll go through these steps a total of 8 times. Apply them to the equation above and you should find that in your example $\chi^2=15.6$.

Once you've calculated $\chi^2$ you use linear interpolation of a $\chi^2$ table to find the value of $p$.

$$p=y_0+(y_1-y_0)\frac{\chi^2-x_0}{x_1-x_0}$$

The degrees of freedom will always be 1 less than the number of observations. You had 8 observations (four sets of clicks & four sets of no-clicks), so in this case there are 7 degrees of freedom. Finally, for confidence interval, $CI = 1-p$. Carrying out your example, $p=0.03$, so Test 3 is the winner with a confidence of 97%.

G-Test

Another cited choice is the G-test. This is closely related and should yield comparable results to the $\chi^2$ test.

$$G=2\sum_i O_i\centerdot ln(\frac{O_i}{E_i})$$

where $O_i$ & $E_i$ are calculated in the same way as above. Repeat this for all 8 of your attributes and you should end up with $G=15.7$. Use the same $\chi^2$ table as above and you'll find that $p=0.029$ and confidence is once again 97%.

Z-Score

There are some articles online that recommend using the Z-score (AKA standard score). I disagree with this approach but include it for the sake of completeness.

$$z=\frac{p-p_c}{\sqrt{\frac{p(1-p)}{N}+\frac{p_c(1-p_c)}{N_c}}}$$

Where $p$ is the conversion rate (online this is referred to as CTR) of one of your elements, $N$ is the number of impressions for that element, $p_c$ and $N_c$ are the CTR & impressions for the control group. To calculate the Z-score for your winning group, we'll use:

$$p_c = 0.24, N_c = 50$$ $$p = 0.566, N = 53$$

Plugging these into the equation above, you'll get $Z=3.58$. As long as $Z>1.96$ you have a result that reaches 95% confidence. If you were using this as your criteria, you'd calculate the Z-score for each of your variants and take the highest.

The reason that I disagree with this approach is that there are situations for which I cannot verify the results via another method. If I use the data listed below:

               Impressions      Clicks
Control                50          12
Variation 1            55          23

I get a Z-score of 1.98, which by that test's criteria shows significance. However, if I try these same values in a $\chi^2$ calculator I find confidence far below 90%. I assume this is because the total number of impressions is low, which intuitively makes sense to me.

I'm not a statistician, so I can't do better than guess at the reason, but this leads me to trust Pearson's $\chi^2$ test more. My choice of the $\chi^2$ test over the G-test is arbitrary. I chose it first and have stuck with it, though I can say that I've also tried the G-test a few times and the results are always comparable.

I hope this helps. I also hope that someone with greater understanding can comment on my methods & choices.

$\endgroup$
2
  • $\begingroup$ How did you calculate E(1c). Can you please elaborate? $\endgroup$
    – Arpit Rai
    Oct 21, 2016 at 12:41
  • $\begingroup$ (Row total * Column total) / total n for the table. You can find a full explanation here. $\endgroup$
    – Lenwood
    Oct 28, 2016 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.