0
$\begingroup$

I work in retail analyzing the results of various marketing campaigns. Many of them are some form of direct messaging (Email, text message, Snail Mail, etc.). When we comb through the database and run the results we calculate a varied number of results. We do this through a standard Treatment vs Holdout scenario, and usually deal with response rates of < 5%.

Response Rate: (# of people who Shopped/# of people contacted) Transactions/Person: (# of Transactions/# of people who shopped) Average Basket: (Total Sales of people who shopped / Total Transactions) Sales/Person: (Total Sales of people who shopped / # of people shopped)

Each of the above can be tested for statistical significance without a problem. However, we start running into problems trying to see if the campaign as a whole had a statistically significant lift in sales. No single average exists that we can extrapolate from, as total sales is, at it's simplest a combination of response rate and sales/person.

We entertained the idea of averaging the Total Sales by # of people sent, but once we do that we end up with a standard deviation built off of a non-normal distribution (most people did not shop so their sales are 0, and those who shopped are well above 0).

So, I'm attempting to figure out some way of determining if the total difference in sales between a test and control group is statistically significant.

Any thoughts?

Improve this question
$\endgroup$
2
  • $\begingroup$ You have two factors that define sales $s_i=r_i\cdot n_i$ for each campaign $i$. You know that properties of $r_i$ and $n_i$. Your problem is that you don't know how to obtain properties of $s_i$ given the above. Am I correctly capturing the essence of the issue? $\endgroup$
    – Aksakal
    Mar 4, 2015 at 18:27
  • $\begingroup$ Correct. I can determine if Ri and/or Ni are signifcant, but not if Si is. In a more general term, we have Yi = X1iX2iX3i.... and can determine any any X properties, but not the Y significance. $\endgroup$
    – Robert Richardson
    Mar 4, 2015 at 18:41

1 Answer 1

Reset to default
0
$\begingroup$

Based on your clarification, I think that your best option is Monte Carlo simulations. It's used a lot in a similar cases, e.g. operational losses modeling in finance. In this example, we have the distribution of loss severity and frequency, but are interested in the loss which is the product of frequency and severity. So, we use simulations.

The trick is to obtain the correlations between the frequency and severity, then we can use copulas or other means to produce consistent scenarios and Monte Carlo simulations. The matter is that your response rate can be correlated to your sales/person, and usually is. That's why you have to incorporate this correlation somehow, maybe with copulas.

This must be similar to your case.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer. I was hoping it wouldn't be simulation, but what can you do? I think I can assume that the basket size is normal, I just need to now look into the proper way to simulate the potential response rate. $\endgroup$
    – Robert Richardson
    Mar 4, 2015 at 18:58
  • $\begingroup$ You'll get used to simulation, it's a standard way of handling this issue in finance. There are many benefits, such as getting the full distribution of sales not just one or two statistics. $\endgroup$
    – Aksakal
    Mar 4, 2015 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.