Permutation test and wilcoxon rank-sum test giving different results

Question

I am comparing the revenue (at user level) between control & test to see if there is a significant difference. Both control & test have roughly about 39k data points . I am doing a permutation test and also a wilcoxon rank-sum test to check for significant difference. From the permutation test, I am getting a p-value of 0.008 and wilcoxon ranksum test gives a pvalue of 0.31. Drastically different results leading to different conclusions. Can someone help me understand why these results are so different ?

Procedure followed for permutation test

1. Calculate the mean revenue for test & control and calculate the difference in mean
2. Concatenate control & test data and permute. Draw samples randomly to create test & control and calculate difference in mean (using numpy.random.permutation)
3. Repeat step-2 for 20,000 times.
4. Calculate the fraction of times (out of 20,000) where difference in mean is at least as big as the one we observed in step 1

Procedure followed for Wilcoxon rank-sum test

1. scipy.stats.ranksums(test,control)

Answer 1

These are different tests; it's not surprising that they give different results.

Your permutation test is a permutation test for differences in the mean -- like a permutation-based t-test. If the test distribution has a larger mean it should reject in favour of the test distribution having a larger mean. If the control distribution has a larger mean it should reject in favour of the test distribution having a larger mean. If two distributions are different but have the same mean, the power will be low and it will be hard to predict which direction it rejects.

The Wilcoxon rank-sum test is not a test for differences in the mean. It is quite possible that the test distribution has a larger mean but the Wilcoxon test rejects in favour of the control distribution values being larger.

You can make a permutation test for any statistic you can compute from the sample. These tests will all be different, in the sense that they will have highest power against different alternatives. The Wilcoxon and mean permutation tests are just two of the many possible examples.