Achieving statistical significance in an A/B test Out of one pool, I've randomly selected a small group (group A) and put everyone else in group B. Then, I'm observing whether or not they take a desired action.
If I'm looking to test at a 95% confidence, what's the best methodology and stats test to use? I read that with a large enough sample, it's ok to use a standard t-test, but can someone tell me if that's accurate?
Assuming it is, I've applied it and come up with the result that group A has a much higher rate of the desired action than group B and applying the error bars, it's still showing that group A > group B (albeit a smaller rate). This is highly unlikely. At worst, the expectation was group A's actions would be more or less the same to group B.
Any comments on what would cause such an unlikely result even with a high confidence level?
 A: The appropriate methodology for statistically testing your data would be a chi squared test on a contingency table. First, put your data in tabular form, which could look like this (I am using R, but any statistics package can do this):
obs <- cbind(c(20,100),c(1000,10000))
rownames(obs) <- c("takes action","takes no action")
colnames(obs) <- c("A","B")
obs

Result:
                  A     B
takes action     20  1000
takes no action 100 10000

Then you do the test:
chisq.test(obs)

Result:
        Pearson's Chi-squared test with Yates' continuity correction

data:  obs 
X-squared = 7.2933, df = 1, p-value = 0.006921

The p value is less than the conventional threshold of 0.05, so your data would be highly unlikely if the groups were in fact identical in their propensity to take the action. The group sizes should not matter for this test (unless there are less than five entries in some cell in the table above, in which case one should use Fisher's Exact Test). And we can calculate quickly that 20% in group A but only 10% in group B take the action.
Now, that is the standard statistical treatment of your data. As to why there is a difference between your groups, that is a question of interpretation, and statistics will only be of limited use here... You will need to look at the exact nature of the action people take or do not take, at the different experiences you expose groups A and B to and so on. After all, if you expose people to different experiences, I assume that you hope to elicit different responses, otherwise why use different experiences, right?
Good luck!
