1
$\begingroup$

I am comparing the revenue (at user level) between control & test to see if there is a significant difference. Both control & test have roughly about 39k data points . I am doing a permutation test and also a wilcoxon rank-sum test to check for significant difference. From the permutation test, I am getting a p-value of 0.008 and wilcoxon ranksum test gives a pvalue of 0.31. Drastically different results leading to different conclusions. Can someone help me understand why these results are so different ?

Procedure followed for permutation test

  1. Calculate the mean revenue for test & control and calculate the difference in mean
  2. Concatenate control & test data and permute. Draw samples randomly to create test & control and calculate difference in mean (using numpy.random.permutation)
  3. Repeat step-2 for 20,000 times.
  4. Calculate the fraction of times (out of 20,000) where difference in mean is at least as big as the one we observed in step 1

Procedure followed for Wilcoxon rank-sum test

  1. scipy.stats.ranksums(test,control)
$\endgroup$
1
  • $\begingroup$ Replace step (2) by "Concatenate the ranks of the control & test data and permute..." $\endgroup$
    – whuber
    Mar 23, 2021 at 21:17

1 Answer 1

2
$\begingroup$

These are different tests; it's not surprising that they give different results.

Your permutation test is a permutation test for differences in the mean -- like a permutation-based t-test. If the test distribution has a larger mean it should reject in favour of the test distribution having a larger mean. If the control distribution has a larger mean it should reject in favour of the test distribution having a larger mean. If two distributions are different but have the same mean, the power will be low and it will be hard to predict which direction it rejects.

The Wilcoxon rank-sum test is not a test for differences in the mean. It is quite possible that the test distribution has a larger mean but the Wilcoxon test rejects in favour of the control distribution values being larger.

You can make a permutation test for any statistic you can compute from the sample. These tests will all be different, in the sense that they will have highest power against different alternatives. The Wilcoxon and mean permutation tests are just two of the many possible examples.

$\endgroup$
2
  • $\begingroup$ Thank you Thomas ! So , you are suggesting to trust the permutation test result in this case? How can I calculate the power of my permutation test ? Or do you suggest any other testing method for this case ? $\endgroup$
    – bp0308
    Mar 23, 2021 at 21:32
  • $\begingroup$ They are both good tests; you need to decide which one you want. The power of the permutation test for the mean will be very close to the power you would calculate for a t-test, so that's the easiest one to do. $\endgroup$ Mar 24, 2021 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.