The material I got only described how to test if there is difference (null hypothesis: H0 = H1).

However, what I want to test is if the test version is doing better than control: null hypothesis: H0 >= H1 .

How to do it?

  • $\begingroup$ What's "h0" and "h1" in "null hypothesis: h0=h1"?? $\endgroup$ – Glen_b Jan 30 '16 at 13:15
  • $\begingroup$ As @Glen_b hints, some confusion of notation here between labels for hypotheses and what they are. I fixed "h" to "H"; otherwise it's important to think that through. $\endgroup$ – Nick Cox Jan 30 '16 at 15:24
  • $\begingroup$ @Nick Actually, besides that likely possibility, I also wondered if perhaps $h$ was intended to represent something else (for example, perhaps $\mu$, for example, since $h$ sort of looks like a flipped $\mu$). $\endgroup$ – Glen_b Jan 30 '16 at 15:38
  • $\begingroup$ thanks for everyone. But I read a paper about using odds to do it. $\endgroup$ – user3516947 Feb 11 '16 at 23:35

One answer is simply to look at the data to see.

Absent real data from the OP, I just steal the fake data given by @Glen_b in a comment (and thereby reinforce his point that the median is not the message; nod to Marshall McLuhan).

A common recipe is two box plots side by side. Here I give a hybrid box and quantile plot in the manner of Emanuel Parzen. That is, all the data points are plotted versus a tacit cumulative probability scale. The letter as well as the spirit of a box plot are honoured: half the data points are inside the box, as every account explains, and half are outside, which some accounts fail to emphasise, perhaps because it seems too obvious.

The medians are the same, by construction, but the distributions are not at all the same.

enter image description here

Another recipe is a quantile-quantile plot. We start with the idea of plotting minimum in sample 1 versus minimum in sample 2, and so on, up to maximum in sample 1 versus maximum in sample 2. Unequal sample sizes do not undermine the idea, as we can just interpolate within the larger sample to get quantiles corresponding to those of the smaller sample. This plot shows that for corresponding quantiles sample 1 is almost always scoring lower than sample 2; the equal medians are the only exception.

enter image description here


The Mann Whitney U Test will tell you if the medians of the datasets are significantly different from each other. In order to tell which dataset is doing better you need to consider what the data is and what you are trying to find out about it.

For example, if you were measuring some error metric where lower is better and your test returns $p<0.05$ then you can compare the medians and conclude that the dataset with the lower median was performing significantly better.

Conversely if you were measuring accuracy (0-100%) where the greater number is more desirable you may conclude that the dataset with the higher median was performing better.

Edit: Thanks to @Glen_b for pointing out a flaw in my answer, while it appears the test will identify a difference in the median...

In practice, the Mann-Whitney U test is more broadly used to interpret whether there are differences in the "distributions" of two groups or differences in the "medians" of two groups.

this paper rightly points out that differences in spread are also important.

  • $\begingroup$ The Mann Whitney U test is not a test of medians. You can easily make samples with identical medians but significant Mann-Whitney statistics. e.g. - sample 1: -100 -80 -60 -40 -20 0 20 40 60 80 100 sample 2: ` -19 -18 -17 -16 -15` -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 5 110 120 130 140 150 160 170 180 190 200 210 220 230 240 Both samples have median 0, but the Mann Whitney test should lead you to reject the null. $\endgroup$ – Glen_b Jan 30 '16 at 13:16
  • $\begingroup$ Apologies you are correct, however, I believe the test is often used in this way: "In practice, the Mann-Whitney U test is more broadly used to interpret whether there are differences in the "distributions" of two groups or differences in the "medians" of two groups." [statistics.laerd.com/premium-sample/mwut/…. I've learned something too! $\endgroup$ – CatsLoveJazz Jan 30 '16 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.