I am comparing the revenue (at user level) between control & test to see if there is a significant difference. Both control & test have roughly about 39k data points . I am doing a permutation test and also a wilcoxon rank-sum test to check for significant difference. From the permutation test, I am getting a p-value of 0.008 and wilcoxon ranksum test gives a pvalue of 0.31. Drastically different results leading to different conclusions. Can someone help me understand why these results are so different ?
Procedure followed for permutation test
- Calculate the mean revenue for test & control and calculate the difference in mean
- Concatenate control & test data and permute. Draw samples randomly to create test & control and calculate difference in mean (using numpy.random.permutation)
- Repeat step-2 for 20,000 times.
- Calculate the fraction of times (out of 20,000) where difference in mean is at least as big as the one we observed in step 1
Procedure followed for Wilcoxon rank-sum test