Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

We have two cohorts of 1000 samples each. We measure 2 quantities on each cohort. The first one is a binary variable. The second is a real number that follows a heavy tail distribution. We want to assess which cohort performs best for each metric. There are plenty of statistical tests to choose from: people suggest z-test, others use t-test, and others Mann–Whitney U.

  • Which test or tests should we choose for each metric for our case?
  • What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?
share|improve this question

migrated from datascience.stackexchange.com Jun 26 at 10:28

This question came from our site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field.

add comment

2 Answers 2

up vote 11 down vote accepted

Given that your two metrics are 1) binary and 2) heavy tailed, you should avoid t-test which assumes normal distributions.

I think Mann-Whitney U is your best choice and should be sufficiently efficient even if your distributions were near-normal.

Regarding your second question:

What happens if one test suggests significant difference between cohorts and some other test suggests non-significant difference?

This is not uncommon if the statistical difference is borderline and the data has "messy" sample distributions. This situation requires the analyst to carefully consider all the assumptions and limitations of each statistical test, and give the most weight to the statistical test which has the least number of violations of assumptions.

Take the assumption of Normal distribution. There are various tests for normality, but that's not the end of the story. Some tests work pretty well on symmetric distributions even if there is some deviation from normality, but don't work well on skew distributions.

As a general rule of thumb, I'd suggest that you should not run any test where any of its assumptions are clearly violated.

EDIT: For the second variable, it might be feasible to transform the variable into one that is normally distributed (or at least close) as long as the transform is order-preserving. You need to have good confidence that the transform yields a normal distribution for both cohorts. If you fit the second variable to log-normal distribution, then a log function transforms it to a normal distribution. But if the distribution is Pareto (power law), then there is no transformation to a normal distribution.

EDIT: As suggested in this comment, you should definitely consider Bayesian Estimation as an alternative to t-testing and other Null Hypothesis Significance Testing (NHST).

share|improve this answer
    
Thanks for the info. I was not clear enough, I have two quantities one of which is binary and another which is a real number following heavy tail distribution. I edited the question to clarify this. –  iliasfl Jun 11 at 1:46
    
Yes, I think I understand. You want to run the test twice, once on the binary variable and once on the real variable (heavy tailed distribution). I'm recommending running the Mann-Whitney U for both. –  MrMeritology Jun 11 at 2:48
add comment

For the real-valued data, you might also want to consider generating your own test statistic based on a bootstrap of your data. This approach tends to produce accurate results when you're dealing with non-normal population distributions, or trying to develop a confidence interval around a parameter that doesn't have a convenient analytic solution. (The former is true in your case. I only mention the latter for context.)

For your real-valued data, you'd do the following:

  1. Pool your two cohorts.
  2. From the pool, sample two groups of 1000 elements, with replacement.
  3. Calculate the difference in sample mean between the two groups.
  4. Repeat steps 2 and 3 a few thousand times to develop a distribution of these differences.

Once you've got that distribution, calculate the difference in means for your actual samples, and calculate a p-value.

share|improve this answer
    
Thanks, so you end up with a distribution, which should be normal with some mean and standard deviation. How would you calculate the p-value from that, and maybe the confidence intervals, in order to decide the winner (if there is one)? –  iliasfl Jun 11 at 22:21
    
The distribution would not necessarily be normal. It would have roughly the distribution of whatever it was sampled from. That is the beauty of using a bootstrap. In any case, you get a p-value by calculating your test statistic from your actual results. I.e. difference of the means of each cohort. Then compare that number to the distribution. The percentile you get is your p-value for a one sided test for difference in mean. –  Nathan Gould Jun 12 at 13:46
3  
What Nathan is describing is also the basis for Bayesian methods of significance testing. I have used (and currently use) the Bayesian Estimation Supersedes the T-Test (BEST) approach. You should look at that framework if you intend on implementing a pooling approach. –  cwharland Jun 12 at 21:14
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.