Statistical significance and winner of multivalent test This should be a fairly straight forward, but I can't seem to figure out the answer. I'm doing some A/B(C/D/E...) testing on a website and measuring impressions and clicks. What method should I be using to determine the statistically significant winner?
This is sample data that I would have. There could be any number of tests, all being displayed in relatively-equal distribution and measuring clicks as a successful result. Such as:
Test #  |  Impressions  | Clicks
  1     |      50       |   12
  2     |      55       |   15
  3     |      53       |   30
  4     |      49       |   22

What algorithm should I be using to determine the winner in a statistically significant way?
 A: If you are trying to relate number of clicks or number of impressions by group, I suggest a count regression model (Poisson or negative binomial, for instance) with "test #" as a categorical independent variable.
A: I'm not a statistician, but I've built a number of tools to evaluate statistical significance of A/B/n tests for inclusion in marketing tools. I'd love to have someone with more experience comment on my answer.
Pearson's $\chi^2$ Test
One popular choice, and the one that produces the most consistent verifiable results is Pearson's Chi-Square Test. The math involved is straight forward and pretty easy to follow.  
$$\chi^2=\sum_i\frac{(O_i-E_i)^2}{E_i}$$
Where $O_i$ is the observed value of clicks or no-clicks. $E_i$ is the observed values for clicks times the percentage of clicks, similar for no-clicks. In your example, Test 1 data is:
$$O_1c=12,  O_1nc=38$$
$$E_1c=19.1, E_1nc=30.9$$
For online A/B/n testing you'll calculate these values for clicks & no actions in each group, so in your example you'll go through these steps a total of 8 times. Apply them to the equation above and you should find that in your example $\chi^2=15.6$. 
Once you've calculated $\chi^2$ you use linear interpolation of a $\chi^2$ table to find the value of $p$.
$$p=y_0+(y_1-y_0)\frac{\chi^2-x_0}{x_1-x_0}$$
The degrees of freedom will always be 1 less than the number of observations. You had 8 observations (four sets of clicks & four sets of no-clicks), so in this case there are 7 degrees of freedom. Finally, for confidence interval, $CI = 1-p$. Carrying out your example, $p=0.03$, so Test 3 is the winner with a confidence of 97%.
G-Test
Another cited choice is the G-test. This is closely related and should yield comparable results to the $\chi^2$ test.
$$G=2\sum_i O_i\centerdot ln(\frac{O_i}{E_i})$$
where $O_i$ & $E_i$ are calculated in the same way as above. Repeat this for all 8 of your attributes and you should end up with $G=15.7$. Use the same $\chi^2$ table as above and you'll find that $p=0.029$ and confidence is once again 97%.
Z-Score
There are some articles online that recommend using the Z-score (AKA standard score). I disagree with this approach but include it for the sake of completeness.
$$z=\frac{p-p_c}{\sqrt{\frac{p(1-p)}{N}+\frac{p_c(1-p_c)}{N_c}}}$$
Where $p$ is the conversion rate (online this is referred to as CTR) of one of your elements, $N$ is the number of impressions for that element, $p_c$ and $N_c$ are the CTR & impressions for the control group. To calculate the Z-score for your winning group, we'll use:
$$p_c = 0.24, N_c = 50$$
$$p = 0.566, N = 53$$
Plugging these into the equation above, you'll get $Z=3.58$. As long as $Z>1.96$ you have a result that reaches 95% confidence. If you were using this as your criteria, you'd calculate the Z-score for each of your variants and take the highest.
The reason that I disagree with this approach is that there are situations for which I cannot verify the results via another method. If I use the data listed below:
               Impressions      Clicks
Control                50          12
Variation 1            55          23

I get a Z-score of 1.98, which by that test's criteria shows significance. However, if I try these same values in a $\chi^2$ calculator I find confidence far below 90%. I assume this is because the total number of impressions is low, which intuitively makes sense to me.
I'm not a statistician, so I can't do better than guess at the reason, but this leads me to trust Pearson's $\chi^2$ test more. My choice of the $\chi^2$ test over the G-test is arbitrary. I chose it first and have stuck with it, though I can say that I've also tried the G-test a few times and the results are always comparable.
I hope this helps. I also hope that someone with greater understanding can comment on my methods & choices.
