Interpreting contradicting results T-Test vs. Mann-Whitney test (2 independent samples)

Question

I want to test if two indepent experimental groups are sign. different in terms of one outcome variable, but face the following questions:

Basic setup

• 2 groups with 200 observations each
• Outcome variable:
  • continuous
  • ranging between 0 and 600
  • very peaked: more than 60% of values are equal to 300
  • Group 1: mean = 221.9807, median = 300
  • Group 2: mean = 239.6396, median = 300

Result of statistical tests

Although the outcome varible violates the normality assumption, I orginally used a two-sample independent t-test based on my large group sizes and the central-limit theorem:

Result of T-test: no signifcant difference (p=0.166).

However, to be on the safe side I also used a nonparmetric test, i.e. the Mann-Whitney U test:

Result: of Mann-Whitney U test significant difference (p=0.023).

Questions

• My interpretation: Mean outcome not sig. different, but distribution of outcome varible differs between groups. Is that correct?
• Could it be a problem that the vast majority of my values are equal to 300? Does the MW test ignore values that are equal to the median?

Answer 1

The original Mann-Whitney test assumes continuous distributions.

If there are many observations and few ties, a normal approximation with a correctly-calculated variance is sufficient.

To deal with heavy ties, the Mann-Whitney test needs to properly deal with the effect those ties have on the distribution of ranks under the null; in some cases the effect can be substantial.

What happens in that situation varies from package to package - some packages don't handle heavy ties well.

The p-value from the t-test may well be suspect.

I'd be inclined to perform a permutation test on the actual set of ranks, or if primary interest focuses on testing for a difference in means, perhaps a permutation test based on the means; this way I don't have to rely on either the t-statistic having a distribution close to the t-distribution under the null, nor on the Mann-Whitney correctly dealing with heavy ties.

Answer 2

The Mann Whitney test and the t-test ask different questions, so they give different answers. The t-test is a test of difference in means, the MW is a test of difference in distributions. In addition (as you noted), the t-test makes assumptions about the distributions that the MW does not.

However, if the vast majority of your values are equal to exactly 300, then I think you might want some other method, but I am not sure what. Clearly, your data is not continuous (or values would not be exactly 300) and the t-test definitely assumes continuous variables. In addition, all those values will have the same rank in the MW test (which may be what you mean by "ignoring medians").

I think you need to think about what exactly you want to test between these two distributions.