# What statistical test to use for A/B test?

We have two cohorts of 1000 samples each. We measure 2 quantities on each cohort. The first one is a binary variable. The second is a real number that follows a heavy tail distribution. We want to assess which cohort performs best for each metric. There are plenty of statistical tests to choose from: people suggest z-test, others use t-test, and others Mann–Whitney U.

• Which test or tests should we choose for each metric for our case?
• What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

Given that your two metrics are 1) binary and 2) heavy tailed, you should avoid t-test which assumes normal distributions.

I think Mann-Whitney U is your best choice and should be sufficiently efficient even if your distributions were near-normal.

What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

This is not uncommon if the statistical difference is borderline and the data has "messy" sample distributions. This situation requires the analyst to carefully consider all the assumptions and limitations of each statistical test, and give the most weight to the statistical test which has the least number of violations of assumptions.

Take the assumption of Normal distribution. There are various tests for normality, but that's not the end of the story. Some tests work pretty well on symmetric distributions even if there is some deviation from normality, but don't work well on skew distributions.

As a general rule of thumb, I'd suggest that you should not run any test where any of its assumptions are clearly violated.

EDIT: For the second variable, it might be feasible to transform the variable into one that is normally distributed (or at least close) as long as the transform is order-preserving. You need to have good confidence that the transform yields a normal distribution for both cohorts. If you fit the second variable to log-normal distribution, then a log function transforms it to a normal distribution. But if the distribution is Pareto (power law), then there is no transformation to a normal distribution.

EDIT: As suggested in this comment, you should definitely consider Bayesian Estimation as an alternative to t-testing and other Null Hypothesis Significance Testing (NHST).

• Thanks for the info. I was not clear enough, I have two quantities one of which is binary and another which is a real number following heavy tail distribution. I edited the question to clarify this. Commented Jun 11, 2014 at 1:46
• Yes, I think I understand. You want to run the test twice, once on the binary variable and once on the real variable (heavy tailed distribution). I'm recommending running the Mann-Whitney U for both.
– MrMeritology
Commented Jun 11, 2014 at 2:48
• Why would a Mann-Whitney be suitable for binary data? Commented Aug 13, 2014 at 5:25
• Mann-Whitney U is effective for non-normal distributions, including discrete distributions with two values (i.e. binary). If all data were binary, then maybe another test would work better. Commented Aug 13, 2014 at 20:24
• Can someone confirm whether this is true?...
– user46925
Commented Mar 14, 2016 at 20:38

For the real-valued data, you might also want to consider generating your own test statistic based on a bootstrap of your data. This approach tends to produce accurate results when you're dealing with non-normal population distributions, or trying to develop a confidence interval around a parameter that doesn't have a convenient analytic solution. (The former is true in your case. I only mention the latter for context.)

For your real-valued data, you'd do the following:

2. From the pool, sample two groups of 1000 elements, with replacement.
3. Calculate the difference in sample mean between the two groups.
4. Repeat steps 2 and 3 a few thousand times to develop a distribution of these differences.

Once you've got that distribution, calculate the difference in means for your actual samples, and calculate a p-value.

• Thanks, so you end up with a distribution, which should be normal with some mean and standard deviation. How would you calculate the p-value from that, and maybe the confidence intervals, in order to decide the winner (if there is one)? Commented Jun 11, 2014 at 22:21
• The distribution would not necessarily be normal. It would have roughly the distribution of whatever it was sampled from. That is the beauty of using a bootstrap. In any case, you get a p-value by calculating your test statistic from your actual results. I.e. difference of the means of each cohort. Then compare that number to the distribution. The percentile you get is your p-value for a one sided test for difference in mean. Commented Jun 12, 2014 at 13:46
• What Nathan is describing is also the basis for Bayesian methods of significance testing. I have used (and currently use) the Bayesian Estimation Supersedes the T-Test (BEST) approach. You should look at that framework if you intend on implementing a pooling approach. Commented Jun 12, 2014 at 21:14

I second @MrMeritology's answer. Actually I was wondering whether the MWU test would be less powerful than the test of independent proportions, since the textbooks I learned from and used to teach said that the MWU can be applied only to ordinal (or interval/ratio) data.

But my simulation results, plotted below, indicate that the MWU test is actually slightly more powerful than the proportion test, while controlling type I error well (at population proportion of group 1=0.50).

The population proportion of group 2 is kept at 0.50. The number of iterations is 10,000 at each point. I repeated the simulation without Yate's correction but the results were the same.

library(reshape)

MakeBinaryData <- function(n1, n2, p1){
y <- c(rbinom(n1, 1, p1),
rbinom(n2, 1, 0.5))
g_f <- factor(c(rep("g1", n1), rep("g2", n2)))
d <- data.frame(y, g_f)
return(d)
}

GetPower <- function(n_iter, n1, n2, p1, alpha=0.05, type="proportion", ...){
if(type=="proportion") {
p_v <- replicate(n_iter, prop.test(table(MakeBinaryData(n1, n1, p1)), ...)$p.value) } if(type=="MWU") { p_v <- replicate(n_iter, wilcox.test(y~g_f, data=MakeBinaryData(n1, n1, p1))$p.value)
}

empirical_power <- sum(p_v<alpha)/n_iter
return(empirical_power)
}

p1_v <- seq(0.5, 0.6, 0.01)
set.seed(1)
power_proptest <- sapply(p1_v, function(x) GetPower(10000, 1000, 1000, x))
power_mwu <- sapply(p1_v, function(x) GetPower(10000, 1000, 1000, x, type="MWU"))

• your result picture is all black Commented Jun 27, 2015 at 9:45

For the binary variable, if you want to compare rates, or proportions, between two samples then you can use a proportion test (for equal sample sizes in your case). This tests the null hypothesis that the two proportions of success are equal.

For the continuous variable, I have nothing new to add to what has already been discussed. The choice of test depends on whether parametric assumptions have been met or not.