I work in retail analyzing the results of various marketing campaigns. Many of them are some form of direct messaging (Email, text message, Snail Mail, etc.). When we comb through the database and run the results we calculate a varied number of results. We do this through a standard Treatment vs Holdout scenario, and usually deal with response rates of < 5%.
Response Rate: (# of people who Shopped/# of people contacted) Transactions/Person: (# of Transactions/# of people who shopped) Average Basket: (Total Sales of people who shopped / Total Transactions) Sales/Person: (Total Sales of people who shopped / # of people shopped)
Each of the above can be tested for statistical significance without a problem. However, we start running into problems trying to see if the campaign as a whole had a statistically significant lift in sales. No single average exists that we can extrapolate from, as total sales is, at it's simplest a combination of response rate and sales/person.
We entertained the idea of averaging the Total Sales by # of people sent, but once we do that we end up with a standard deviation built off of a non-normal distribution (most people did not shop so their sales are 0, and those who shopped are well above 0).
So, I'm attempting to figure out some way of determining if the total difference in sales between a test and control group is statistically significant.