Difference in mean of two non normal heavily right skewed samples I have researched quite a bit to try an answer on my own but found very contradicting answers.. Would appreciate if this community can help..
I set up a test to measure the impact of an offer sent via email to customers in the test group.. Three months have passed and my test group constitutes ~ 164K customers, while control has ~ 18K customers (90:10 split). Now I am trying to determine the impact of the offer on test group.. I want to determine if mean number of transactions of test is significantly higher than control.
The problem I have is most of the customers ~95% do not do any transactions in the three months period which makes the distribution heavily right skewed. Which test/methodology can i use to determine if the mean of test > control.. The two sample parametric test has an assumption that the populations should be normally distributed which is so not true in my case and I've read in few places that Mann- Whitney shouldn't be used for comparing means. Plz advise
 A: In Web testing, given your sample size and provided your responses are not extremely skewed, the sample means can still be approximately normally distributed, and Welch's $t$-test (for unequal variance and sample size) can still be applied. You won't get an exact p-value due to violation of assumptions on $t$-distributed random variables, but the approximation should be good enough.
There is no clear cut answer for what constitute extremely skewed responses. This question provides a case of an extremely skewed data, which is unlikely your case given your business metric is "mean number of transactions [per customer]". It might be the case if your metric is "mean spend [per customer]". The answer to this question (by myself) provided a rule of thumb by Kohavi et al. (2014) on what kind of skewness can a $t$-test deal with.
If in doubt, the community's suggestion is to run some simulations to see how the sample mean behaves. One option under such approach is to simulate the distribution of the sample means by bootstraping, and then compare the sample mean distributions (not the distribution for the two set of responses) using Mann-Whitney. A note of caution that if you decided to use Mann-Whitney in this scenario, the result will be dependent on the number of bootstrap samples you have, not the number of original samples.
