Difference between G-test and t-test and which should be used for A/B testing?

Question

The G-Test is a way to get quick estimates of a chi squared distribution, and is recommended by the author of this well-known A/B test tutorial.

This tool assumes a normal distribution and uses difference of means to compute confidence.

What is the difference between a G test and a T test? What are the benefits or downsides to using each method to measure the effectiveness of our A/B tests?

I'm trying to figure out which one I should use to measure the results of my A/B test framework. Our framework has two general use cases: split the group of visitors evenly, show each one a different feature and measure their conversion on some other page (say, the sign up page); and split the group of visitors into the control group (90%) and an experimental group (10%) for a test, and measure conversions on some other page.

Our website gets between 1000 and 200,000 visits per day. These visits are split with an exponential distribution across about 300 pages.

Thanks, Kevin

Answer 1

In general, the test which is less approximate in calculating the test statistics is better, although all will converge to the same results with increasing sample size.

So, since A/B-tests generally focus on binary outcomes, ...

Short answer:

Use the G-test, because it is less approximate.

Long answer:

The t-test, in A/B-tests the case of unequal sample sizes and unequal variance, approximates the difference of two distributions with a t-distribution, which is questionable itself. The two distributions may be unknown, but it is considered that their mean and variance is sufficient to describe it (otherwise any conclusion won't help much), which is of course true for the normal distribution.

In the special case of binary outcome, the binomial distribution can be approximated with a normal distribution with $\mu=np,\sigma^2=np(1-p)$, which is valid for $n*p*(1-p)\geq9$ (rule of the thumb, $n$=trials,$p$=success-rate).

So, in summary, although it is ok to apply the t-test, two approximations are performed to transform the binomial case to a more generic case, which is not necessary here, since less approximative tests like the G-test or (even better) Fisher's exact test are available for this special case. The Fisher's exact test should be applied especially if the sample-size is less equal 20 (another rule of the thumb), but I guess this does not matter in a solid A/B-test.

Answer 2

Ben Tilly's page that you referenced is an excellent summary of A/B testing for beginners. As you get into more detailed questions/study design problems, however, it is worth seeking out more detailed primary sources. Kohavi et al published a seminal paper on AB testing that is a good combination of comprehensiveness and readability. I highly recommend it: http://exp-platform.com/Documents/GuideControlledExperiments.pdf.

Back to your questions, the real questions you should be asking yourself are:

1. How many impressions do I need to get in the treatments and control for the result to be significantly significant?
2. What is the minimum effect size that I am concerned with? Are you interested in treatments that are at least 5% better than controls, or .005% better?
3. In case of multiple treatments, is there a scenario for comparing treatments to each other, or is it sufficient to compare each treatment to the control?
4. What variables are important to measure to ensure that treatment groups are not affected by unintentional side effects of your experiment. Kohavi paper has a great example of this in terms of web site performance: if your treatment experience is slower then control for whatever reason (more images, different server, quick-and-dirty code), this has potential to seriously derail the test.
5. Does it make more sense to enroll users or impressions into the experiments? In other words, does it make sense to ensure that the user always gets either control or treatment experience for the duration of session/trial period, or can you enroll each page impression into the test independently?

As you work through these questions, you will eventually end up with a better understanding of test parameters. Combined with your domain knowledge (e.g. whether your site experiences a strong cyclical pattern that you would like to control for), appetite for exposing users to experiments (are you actually willing to show the treatment experience to many users, or would you rather contain the potential damage) and desired speed of obtaining results, this understanding will guide you towards ultimately determining how to split up the overall traffic among controls and treatments.

I hate answering specific questions with "it depends", but in this case it really does depend on what is going on with your site and experiment. Under certain condition, it will not make a significant difference whether to split the traffic 50/50 or 90/10, while in different circumstances this may be very important. YMMV, but a good reference like the paper cited above will definitely move you in the right direction.

Answer 3

I can't comment on the original post since I lack StackExchange points or whatever, but I just wanted to point out that for the p-value, ABBA doesn't use a simple normal-approximation-based Z-test, though I can see how you might think that from a brief read of the page. ABBA uses exact binomial statistics up to sample size 100, beyond that it does rely on the normal approximation with a continuity correction. I haven't seen cases where it differs greatly from "less approximate" tests but I would be very interested in seeing any such cases if you run into them.

There are no t-distributions or t-tests present in any case.

For confidence intervals, it does always rely on a normal approximation, though it uses the Agresti-Coull method which performs pretty well.