I am using the Wilcoxon Rank-Sum/Mann-Whitney U test to compare metrics between two groups. In almost all cases, I get huge values for the statistic and only p-values around zero. In one case I got reasonable values for p around 0.22, which raises the question if the other cases are actually around zero. The number of values in the distributions I compare ranges between 200k and 800k depending on the metric and can differ between the two groups.
The null hypothesis is that both groups are from the same distribution.
- Is it possible to get p-values around zero with these tests and what does that mean?
- Is there a cause which could explain this behavior?
Please comment if you need more information on my setup.
The statistic values seem to be reasonable as they seem to be around the expected mean of 1/2 the product of the size of the two groups.
It seems that I have very few very high outliers but they should not be an issue with the chosen test as described here: Do we need to worry about outliers when using rank-based tests?.