# What statistical test to use for A/B test?

We have two cohorts of 1000 samples each. We measure 2 quantities on each cohort. The first one is a binary variable. The second is a real number that follows a heavy tail distribution. We want to assess which cohort performs best for each metric. There are plenty of statistical tests to choose from: people suggest z-test, others use t-test, and others Mann–Whitney U.

• Which test or tests should we choose for each metric for our case?
• What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

Given that your two metrics are 1) binary and 2) heavy tailed, you should avoid t-test which assumes normal distributions.

I think Mann-Whitney U is your best choice and should be sufficiently efficient even if your distributions were near-normal.

What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

This is not uncommon if the statistical difference is borderline and the data has "messy" sample distributions. This situation requires the analyst to carefully consider all the assumptions and limitations of each statistical test, and give the most weight to the statistical test which has the least number of violations of assumptions.

Take the assumption of Normal distribution. There are various tests for normality, but that's not the end of the story. Some tests work pretty well on symmetric distributions even if there is some deviation from normality, but don't work well on skew distributions.

As a general rule of thumb, I'd suggest that you should not run any test where any of its assumptions are clearly violated.

EDIT: For the second variable, it might be feasible to transform the variable into one that is normally distributed (or at least close) as long as the transform is order-preserving. You need to have good confidence that the transform yields a normal distribution for both cohorts. If you fit the second variable to log-normal distribution, then a log function transforms it to a normal distribution. But if the distribution is Pareto (power law), then there is no transformation to a normal distribution.

EDIT: As suggested in this comment, you should definitely consider Bayesian Estimation as an alternative to t-testing and other Null Hypothesis Significance Testing (NHST).

• Thanks for the info. I was not clear enough, I have two quantities one of which is binary and another which is a real number following heavy tail distribution. I edited the question to clarify this. – iliasfl Jun 11 '14 at 1:46
• Yes, I think I understand. You want to run the test twice, once on the binary variable and once on the real variable (heavy tailed distribution). I'm recommending running the Mann-Whitney U for both. – MrMeritology Jun 11 '14 at 2:48
• Why would a Mann-Whitney be suitable for binary data? – Glen_b -Reinstate Monica Aug 13 '14 at 5:25
• Mann-Whitney U is effective for non-normal distributions, including discrete distributions with two values (i.e. binary). If all data were binary, then maybe another test would work better. – MrMeritology Aug 13 '14 at 20:24
• Can someone confirm whether this is true?... – user46925 Mar 14 '16 at 20:38

For the real-valued data, you might also want to consider generating your own test statistic based on a bootstrap of your data. This approach tends to produce accurate results when you're dealing with non-normal population distributions, or trying to develop a confidence interval around a parameter that doesn't have a convenient analytic solution. (The former is true in your case. I only mention the latter for context.)

For your real-valued data, you'd do the following:

2. From the pool, sample two groups of 1000 elements, with replacement.
3. Calculate the difference in sample mean between the two groups.
4. Repeat steps 2 and 3 a few thousand times to develop a distribution of these differences.

Once you've got that distribution, calculate the difference in means for your actual samples, and calculate a p-value.

• Thanks, so you end up with a distribution, which should be normal with some mean and standard deviation. How would you calculate the p-value from that, and maybe the confidence intervals, in order to decide the winner (if there is one)? – iliasfl Jun 11 '14 at 22:21
• The distribution would not necessarily be normal. It would have roughly the distribution of whatever it was sampled from. That is the beauty of using a bootstrap. In any case, you get a p-value by calculating your test statistic from your actual results. I.e. difference of the means of each cohort. Then compare that number to the distribution. The percentile you get is your p-value for a one sided test for difference in mean. – Nathan Gould Jun 12 '14 at 13:46
• What Nathan is describing is also the basis for Bayesian methods of significance testing. I have used (and currently use) the Bayesian Estimation Supersedes the T-Test (BEST) approach. You should look at that framework if you intend on implementing a pooling approach. – cwharland Jun 12 '14 at 21:14

I second @MrMeritology's answer. Actually I was wondering whether the MWU test would be less powerful than the test of independent proportions, since the textbooks I learned from and used to teach said that the MWU can be applied only to ordinal (or interval/ratio) data.

But my simulation results, plotted below, indicate that the MWU test is actually slightly more powerful than the proportion test, while controlling type I error well (at population proportion of group 1=0.50). The population proportion of group 2 is kept at 0.50. The number of iterations is 10,000 at each point. I repeated the simulation without Yate's correction but the results were the same.

library(reshape)

MakeBinaryData <- function(n1, n2, p1){
y <- c(rbinom(n1, 1, p1),
rbinom(n2, 1, 0.5))
g_f <- factor(c(rep("g1", n1), rep("g2", n2)))
d <- data.frame(y, g_f)
return(d)
}

GetPower <- function(n_iter, n1, n2, p1, alpha=0.05, type="proportion", ...){
if(type=="proportion") {
p_v <- replicate(n_iter, prop.test(table(MakeBinaryData(n1, n1, p1)), ...)$p.value) } if(type=="MWU") { p_v <- replicate(n_iter, wilcox.test(y~g_f, data=MakeBinaryData(n1, n1, p1))$p.value)
}

empirical_power <- sum(p_v<alpha)/n_iter
return(empirical_power)
}

p1_v <- seq(0.5, 0.6, 0.01)
set.seed(1)
power_proptest <- sapply(p1_v, function(x) GetPower(10000, 1000, 1000, x))
power_mwu <- sapply(p1_v, function(x) GetPower(10000, 1000, 1000, x, type="MWU"))

• your result picture is all black – Ooker Jun 27 '15 at 9:45