I was running a permutation test on my data. My goal was to test difference in means in two samples, by resampling one of them.
I got results like these:
From my understanding, the p-value here is the proportion of data points in the sample that have bigger mean difference in the permutation groups compared to the complete data. Thus, a p-value closer to 1 tells me that 100% of the data in the permuted samples have bigger mean difference, and the p-value of 0.02 tells me that only 2% of the data in the permuted samples have bigger mean difference. In the second I reject the null hypothesis, but in the first case I do not. My question is: why?
Why 90%, 95%, 99%... of data in the permuted samples having bigger difference from the actual mean difference does not tell me to reject the null hypothesis? I mean, the permuted groups seems to have really significant differences in means than the actual one for the whole data, but I do not reject the null hypothesis. When the p-value is as low as 0.05, I can visualize that there are a lot of difference in the permuted samples comparing to the actual mean. But in only one case I reject the null hypothesis. This is quite confusing.