What statistical test to use for A/B test?

Question

We have two cohorts of 1000 samples each. We measure 2 quantities on each cohort. The first one is a binary variable. The second is a real number that follows a heavy tail distribution. We want to assess which cohort performs best for each metric. There are plenty of statistical tests to choose from: people suggest z-test, others use t-test, and others Mann–Whitney U.

• Which test or tests should we choose for each metric for our case?
• What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

Answer 1

Given that your two metrics are 1) binary and 2) heavy tailed, you should avoid t-test which assumes normal distributions.

I think Mann-Whitney U is your best choice and should be sufficiently efficient even if your distributions were near-normal.

Regarding your second question:

What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

This is not uncommon if the statistical difference is borderline and the data has "messy" sample distributions. This situation requires the analyst to carefully consider all the assumptions and limitations of each statistical test, and give the most weight to the statistical test which has the least number of violations of assumptions.

Take the assumption of Normal distribution. There are various tests for normality, but that's not the end of the story. Some tests work pretty well on symmetric distributions even if there is some deviation from normality, but don't work well on skew distributions.

As a general rule of thumb, I'd suggest that you should not run any test where any of its assumptions are clearly violated.

EDIT: For the second variable, it might be feasible to transform the variable into one that is normally distributed (or at least close) as long as the transform is order-preserving. You need to have good confidence that the transform yields a normal distribution for both cohorts. If you fit the second variable to log-normal distribution, then a log function transforms it to a normal distribution. But if the distribution is Pareto (power law), then there is no transformation to a normal distribution.

EDIT: As suggested in this comment, you should definitely consider Bayesian Estimation as an alternative to t-testing and other Null Hypothesis Significance Testing (NHST).

Answer 2

For the real-valued data, you might also want to consider generating your own test statistic based on a bootstrap of your data. This approach tends to produce accurate results when you're dealing with non-normal population distributions, or trying to develop a confidence interval around a parameter that doesn't have a convenient analytic solution. (The former is true in your case. I only mention the latter for context.)

For your real-valued data, you'd do the following:

1. Pool your two cohorts.
2. From the pool, sample two groups of 1000 elements, with replacement.
3. Calculate the difference in sample mean between the two groups.
4. Repeat steps 2 and 3 a few thousand times to develop a distribution of these differences.

Once you've got that distribution, calculate the difference in means for your actual samples, and calculate a p-value.

Answer 3

I second @MrMeritology's answer. Actually I was wondering whether the MWU test would be less powerful than the test of independent proportions, since the textbooks I learned from and used to teach said that the MWU can be applied only to ordinal (or interval/ratio) data.

But my simulation results, plotted below, indicate that the MWU test is actually slightly more powerful than the proportion test, while controlling type I error well (at population proportion of group 1=0.50).

The population proportion of group 2 is kept at 0.50. The number of iterations is 10,000 at each point. I repeated the simulation without Yate's correction but the results were the same.

library(reshape)

MakeBinaryData <- function(n1, n2, p1){
  y <- c(rbinom(n1, 1, p1), 
        rbinom(n2, 1, 0.5))
  g_f <- factor(c(rep("g1", n1), rep("g2", n2)))
  d <- data.frame(y, g_f)
  return(d)
}

GetPower <- function(n_iter, n1, n2, p1, alpha=0.05, type="proportion", ...){
  if(type=="proportion") {
    p_v <- replicate(n_iter, prop.test(table(MakeBinaryData(n1, n1, p1)), ...)$p.value)
  }

  if(type=="MWU") {
    p_v <- replicate(n_iter, wilcox.test(y~g_f, data=MakeBinaryData(n1, n1, p1))$p.value)
  }

  empirical_power <- sum(p_v<alpha)/n_iter
  return(empirical_power)
}

p1_v <- seq(0.5, 0.6, 0.01)
set.seed(1)
power_proptest <- sapply(p1_v, function(x) GetPower(10000, 1000, 1000, x))
power_mwu <- sapply(p1_v, function(x) GetPower(10000, 1000, 1000, x, type="MWU"))