# Statistical signficance of aggregated terms

I work in retail analyzing the results of various marketing campaigns. Many of them are some form of direct messaging (Email, text message, Snail Mail, etc.). When we comb through the database and run the results we calculate a varied number of results. We do this through a standard Treatment vs Holdout scenario, and usually deal with response rates of < 5%.

Response Rate: (# of people who Shopped/# of people contacted) Transactions/Person: (# of Transactions/# of people who shopped) Average Basket: (Total Sales of people who shopped / Total Transactions) Sales/Person: (Total Sales of people who shopped / # of people shopped)

Each of the above can be tested for statistical significance without a problem. However, we start running into problems trying to see if the campaign as a whole had a statistically significant lift in sales. No single average exists that we can extrapolate from, as total sales is, at it's simplest a combination of response rate and sales/person.

We entertained the idea of averaging the Total Sales by # of people sent, but once we do that we end up with a standard deviation built off of a non-normal distribution (most people did not shop so their sales are 0, and those who shopped are well above 0).

So, I'm attempting to figure out some way of determining if the total difference in sales between a test and control group is statistically significant.

Any thoughts?

• You have two factors that define sales $s_i=r_i\cdot n_i$ for each campaign $i$. You know that properties of $r_i$ and $n_i$. Your problem is that you don't know how to obtain properties of $s_i$ given the above. Am I correctly capturing the essence of the issue? Mar 4, 2015 at 18:27
• Correct. I can determine if Ri and/or Ni are signifcant, but not if Si is. In a more general term, we have Yi = X1iX2iX3i.... and can determine any any X properties, but not the Y significance. Mar 4, 2015 at 18:41