# Achieving statistical significance in an A/B test

Out of one pool, I've randomly selected a small group (group A) and put everyone else in group B. Then, I'm observing whether or not they take a desired action.

If I'm looking to test at a 95% confidence, what's the best methodology and stats test to use? I read that with a large enough sample, it's ok to use a standard t-test, but can someone tell me if that's accurate?

Assuming it is, I've applied it and come up with the result that group A has a much higher rate of the desired action than group B and applying the error bars, it's still showing that group A > group B (albeit a smaller rate). This is highly unlikely. At worst, the expectation was group A's actions would be more or less the same to group B.

Any comments on what would cause such an unlikely result even with a high confidence level?

• I think I am missing something. If you randomly selected group A from group B, then the groups should not be different, right? So I assume that you in fact did not select at random, but based on some criterion. How large are your groups A and B? Apart from that, a t test is not appropriate for yes/no variables (take an action or not), a contingency table and a chi squared test would be better. Commented Jan 10, 2013 at 9:09
• Think of a typical website A/B test. I put a percentage of visitors in A and everyone else in B. Group A sees one experience. Group B sees another experience. Then, I measure something like..do they click this button. I want to measure the rate at which they click the button and see what the difference is between those two groups. One group is much smaller than the other, like 1 in group A for every 20 in group B. But, the people who are selected to be in each group are selected randomly. Commented Jan 10, 2013 at 20:30

The appropriate methodology for statistically testing your data would be a chi squared test on a contingency table. First, put your data in tabular form, which could look like this (I am using R, but any statistics package can do this):

obs <- cbind(c(20,100),c(1000,10000))
rownames(obs) <- c("takes action","takes no action")
colnames(obs) <- c("A","B")
obs


Result:

                  A     B
takes action     20  1000
takes no action 100 10000


Then you do the test:

chisq.test(obs)


Result:

        Pearson's Chi-squared test with Yates' continuity correction

data:  obs
X-squared = 7.2933, df = 1, p-value = 0.006921


The p value is less than the conventional threshold of 0.05, so your data would be highly unlikely if the groups were in fact identical in their propensity to take the action. The group sizes should not matter for this test (unless there are less than five entries in some cell in the table above, in which case one should use Fisher's Exact Test). And we can calculate quickly that 20% in group A but only 10% in group B take the action.

Now, that is the standard statistical treatment of your data. As to why there is a difference between your groups, that is a question of interpretation, and statistics will only be of limited use here... You will need to look at the exact nature of the action people take or do not take, at the different experiences you expose groups A and B to and so on. After all, if you expose people to different experiences, I assume that you hope to elicit different responses, otherwise why use different experiences, right?

Good luck!