4
$\begingroup$

I am testing Mann-Whitney rank test with two vectors a and b. Vectors are almost similar so I expect a p-value near 0, but the returned p-value is near 1. What is the reason? I read the manual and also run the code with different parameters, but don't get anything near to what I expect.

from scipy.stats import mannwhitneyu
import operator
import numpy as np

a = [1000,100,10,1,10,100,1000,10000,1000,100,10,1]
b = [999,100,10,1,10,100,1000,10000,1000,100,10,1]

print(mannwhitneyu(a, b))

The output of the code:

MannwhitneyuResult(statistic=73.5, pvalue=0.95291544831453867)

Edit:

Let's formalize the problem to eliminate any misunderstanding (please edit the question if something is wrong):

What I try to prove is that rank distribution of data are approximately equal,

Null hypothesis ($H_0$)= "ranked distribution of a and b are approximately equal"

Alternative hypothesis ($H_a$)= "ranked distribution of a and b are not equal"

significance level (alpha) = 0.05

p-value= 0.95

(p-value > alpha) so there is no sufficient evidence that $H_a$ is correct, but we cannot also conclude that ($H_0$) is true.

Instead if (p-value < alpha) was true, then we would have enough evidence against H0 and Ha can be accepted.

So what happened here is that I could not disprove the null hypothesis. However, it doesn't mean that null hypothesis is false. It is like an investigation to accuse MR X of being guilty:

$H_0$=" MR X guilty" $H_a$=" MR X not guilty"

We guess he is not guilty, but we don't have enough proof against him: (p-value > alpha), but that doesn't mean that ($H_0$) would not be true. If we could obtain enough evidence and state (p-value < alpha), then we disprove $H_0$ and we can conclude that Ha is true and he is not guilty.

$\endgroup$
  • 2
    $\begingroup$ I think the p-value is correct. Simply put the Mann-Whitney U tests whether the null hypothesis "ranked distribution of a and b are approximately equal" could be true. As they are almost equal, the probability of this being true is very high or based on your data probability of this being true is high => p-value is high => Null hypothesis cannot be rejected. $\endgroup$ – IWS Jan 31 '17 at 9:51
  • $\begingroup$ Great explanation! so, can we say: p-value for alternative hypothesis is 0.95 => p-value for null hypothesis is 0.05 ? $\endgroup$ – Woeitg Jan 31 '17 at 10:17
  • $\begingroup$ No, you can't say that $\endgroup$ – IWS Jan 31 '17 at 10:31
  • $\begingroup$ @IWS no, that's not what a p-value is. It's definitely not the probability that the null hypothesis is true. $\endgroup$ – Glen_b -Reinstate Monica Jan 31 '17 at 10:33
  • $\begingroup$ @Glen_b true, I edited my comment $\endgroup$ – IWS Jan 31 '17 at 10:34
9
$\begingroup$

The p-value represents the probability of getting a test-statistic at least as extreme$^\dagger$ as the one you had in your sample, if the null hypothesis were true.

A high p-value indicates you saw something really consistent with the null hypothesis (e.g. tossing 151 heads in 300 tosses of a coin you're examining for fairness), and something that's really consistent with the null being true would not cause you to think it was false. (In some situations it might perhaps lead you to think more carefully about the assumptions.)

If you thought that a and b were very similar in values then you'd expect to obtain a high p-value, not a low one. (If you expected a low p-value, you may have some misunderstanding of how p-values work.)

That is, the p-value from the test is consistent with your statement about them being very similar*.

Low p-values are the things that would cause you to hold doubt about the null.


A caveat: the fact that the values move together in pairs across many orders of magnitude indicates a very high correlation, so the assumption of independence is untenable (the data seem to be paired). It should surely raise doubts about the suitability of the test. (I presume you made up the values to see how the test behaves, but if that's real data you have a problem.)


$\dagger$ away from what you'd expect to see under the null, in the direction of what you'd see under the alternative

* however note that values may in some circumstances be very similar (at least in some sense) without the p-value being high.

$\endgroup$
  • $\begingroup$ Thanks or explanation, I misunderstood the concept. Now I edit the question and try to formalize the problem so that hopefully now I understood the explanation of the most confusing concept in statistics! $\endgroup$ – Woeitg Jan 31 '17 at 11:38
  • $\begingroup$ Now my question is, what is the right test in your opinion for this sample? I chose Mann-Whitney U test because I though it is a non-parametric test and we are not risking of selecting the wrong test because there is no assumption on distribution of samples. $\endgroup$ – Woeitg Jan 31 '17 at 11:41
  • $\begingroup$ @Woetig You will have to decided, whether your data are independent or dependend data, suggesting a paired or unpaired test. Have you considered, whether a rank sum test or a signed rank sum test is appropriate? This is not about parametric/non-parametric but a different axis in the choice of tests. $\endgroup$ – Bernhard Jan 31 '17 at 11:43
  • $\begingroup$ good point. I asked it in a different question: stats.stackexchange.com/questions/259126/… $\endgroup$ – Woeitg Jan 31 '17 at 12:09
  • 1
    $\begingroup$ sure. I asked the new question separately. I edited my question with my own interpretation of p-value and analysis. also I add the link to old question. $\endgroup$ – Woeitg Jan 31 '17 at 12:27
2
$\begingroup$

If you are trying to prove that the two vectors are approximately equal then you have two issues: 1) What exactly do you mean by "equal"? and 2) No usual test is appropriate.

For 1) you have to consider if the data are independent or not and whether you want to test means or medians or whatever. For 2) You should look into tests of equivalence such as TOST (two one sided t-tests) which allow you to test the hypothesis you have in mind.

$\endgroup$
  • $\begingroup$ Thanks for answer. Can you elaborate more on your 2nd point maybe with formalizing my question using Null hypothesis? $\endgroup$ – Woeitg Jan 31 '17 at 12:30
  • 1
    $\begingroup$ about your 1st point, I want to compare if underlying distribution of data is the same. I asked another question about this point with more details: stats.stackexchange.com/questions/259126/… $\endgroup$ – Woeitg Jan 31 '17 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.