1
$\begingroup$

I was running a permutation test on my data. My goal was to test difference in means in two samples, by resampling one of them.

I got results like these:

enter image description here

And

enter image description here

From my understanding, the p-value here is the proportion of data points in the sample that have bigger mean difference in the permutation groups compared to the complete data. Thus, a p-value closer to 1 tells me that 100% of the data in the permuted samples have bigger mean difference, and the p-value of 0.02 tells me that only 2% of the data in the permuted samples have bigger mean difference. In the second I reject the null hypothesis, but in the first case I do not. My question is: why?

Why 90%, 95%, 99%... of data in the permuted samples having bigger difference from the actual mean difference does not tell me to reject the null hypothesis? I mean, the permuted groups seems to have really significant differences in means than the actual one for the whole data, but I do not reject the null hypothesis. When the p-value is as low as 0.05, I can visualize that there are a lot of difference in the permuted samples comparing to the actual mean. But in only one case I reject the null hypothesis. This is quite confusing.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Are you sure the p-value in the first plot is not just a display problem? e.g. 1.00E-3 truncated badly could be displayed as 1.00 $\endgroup$
    – mkt
    Commented Aug 2, 2022 at 13:41
  • $\begingroup$ Another possibility is that you are doing a one-sided test $\endgroup$
    – mkt
    Commented Aug 2, 2022 at 13:42
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Aug 2, 2022 at 14:18
  • $\begingroup$ It seems like the two figures are using different data sets. Is that correct? Why is it confusing that the p-value would be different for two different datasets? $\endgroup$
    – John L
    Commented Aug 2, 2022 at 14:56
  • $\begingroup$ both figures are using different datasets but my problem is to interpret the p-value: why a p close to 1 does not reject the null hypothesis and a p close to 0 does in a permutation test?Both of them are telling me that the difference in means in the permutation group are very different to the difference in means in the whole data,one of them tells me that something close to 100% are greater in the permutation group, other tells me that something close to 0% are less in the permutation group $\endgroup$
    – Dimitri
    Commented Aug 2, 2022 at 15:10

1 Answer 1

Reset to default
1
$\begingroup$

I see two possibilities, of which one has been ruled out by OP in comments.

  1. The p-value in the first plot is displayed incorrectly e.g. 1.00E-3 truncated badly could be displayed as 1.00. [OP says this is not the case]
  2. The p-values shown are for one-sided tests that the sample mean is larger than the reference group mean. This would explain both the low p-value in the second plot and the very high p-value in the first plot.
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ When you post your first comment two days ago about the possibility of this being a one-sided test I was not sure about it, but now I came back to the problem and the code and yes, it is a one-sided test. I am learning these kinds of problems, thus some details may be not quite clear to me. But yes, it is a one-sided test and I was testing if the sample mean was lerger than the reference, it is clear now what it means and why the p-values are shown that way. Not a difficult problem but it wasn't clear in the beginning. Thank you very much $\endgroup$
    – Dimitri
    Commented Aug 4, 2022 at 14:48
  • $\begingroup$ @Dimitri Happy to have helped, and I'm glad this solved the problem! $\endgroup$
    – mkt
    Commented Aug 4, 2022 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.