2
$\begingroup$

I want to test if two indepent experimental groups are sign. different in terms of one outcome variable, but face the following questions:

Basic setup

  • 2 groups with 200 observations each
  • Outcome variable:
    • continuous
    • ranging between 0 and 600
    • very peaked: more than 60% of values are equal to 300
    • Group 1: mean = 221.9807, median = 300
    • Group 2: mean = 239.6396, median = 300

Result of statistical tests

Although the outcome varible violates the normality assumption, I orginally used a two-sample independent t-test based on my large group sizes and the central-limit theorem:

Result of T-test: no signifcant difference (p=0.166).

However, to be on the safe side I also used a nonparmetric test, i.e. the Mann-Whitney U test:

Result: of Mann-Whitney U test significant difference (p=0.023).

Questions

  • My interpretation: Mean outcome not sig. different, but distribution of outcome varible differs between groups. Is that correct?
  • Could it be a problem that the vast majority of my values are equal to 300? Does the MW test ignore values that are equal to the median?
$\endgroup$
2
$\begingroup$

The original Mann-Whitney test assumes continuous distributions.

If there are many observations and few ties, a normal approximation with a correctly-calculated variance is sufficient.

To deal with heavy ties, the Mann-Whitney test needs to properly deal with the effect those ties have on the distribution of ranks under the null; in some cases the effect can be substantial.

What happens in that situation varies from package to package - some packages don't handle heavy ties well.

The p-value from the t-test may well be suspect.

I'd be inclined to perform a permutation test on the actual set of ranks, or if primary interest focuses on testing for a difference in means, perhaps a permutation test based on the means; this way I don't have to rely on either the t-statistic having a distribution close to the t-distribution under the null, nor on the Mann-Whitney correctly dealing with heavy ties.

$\endgroup$
1
$\begingroup$

The Mann Whitney test and the t-test ask different questions, so they give different answers. The t-test is a test of difference in means, the MW is a test of difference in distributions. In addition (as you noted), the t-test makes assumptions about the distributions that the MW does not.

However, if the vast majority of your values are equal to exactly 300, then I think you might want some other method, but I am not sure what. Clearly, your data is not continuous (or values would not be exactly 300) and the t-test definitely assumes continuous variables. In addition, all those values will have the same rank in the MW test (which may be what you mean by "ignoring medians").

I think you need to think about what exactly you want to test between these two distributions.

$\endgroup$
  • $\begingroup$ Originally, I wanted to test two hypothesis: 1. Does the distribution of the outcome variable differ between the two conditions / does the experimental variation between the two groups have a significant effect on the distribution of the outcome varible? 2. Does the exerimental variation between the two groups lead to signficant change in the mean of the outcome variable? Why is the outcome variable not continuous although it takes values between 0 and 600? $\endgroup$ – Tim H Jul 9 '15 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.