2
$\begingroup$

I have several markets across the US where a marketing program was launched, and I want to compare the mean weekly unit sales before and after the launch. I'm using the 10-weeks prior to launch as my baseline and the 10 weeks post-launch as my result. Now at the end I will use a paired t-test to compare the change relative to the control markets (not in the program).

However my question is for the interim. Management wants to see updated results every week.

  1. Is it valid to recompute statistical significance each week while the program is still running?
  2. Which test is appropriate? The samples are linked so it should be a paired t-test, but that requires the same observations which I don't have until the end.
Improve this question
$\endgroup$
0
$\begingroup$

The pairing factor is the store within each region (OP later clarified). I would recommend you decide how to boil down the sales figures pre-launch into a single number from which to compare the post-launch figures against. I'll take the simplest, and therefore most interpretable approach here.

  1. It is OK to recompute every week with fresh results. This depends on 2.
  2. Take a close look at the 10 weeks prior to the launch, across all your test and control markets. Are there trends due to external factors? Make sure to adjust for these factors when simplifying to one pre-launch figure per store, whether it's an average of the 10 weeks or the week just before launch or something other adjusted number. Do a paired t-test for each market using store as the pairing factor, then present week-by-week post-launch t-test results to management.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Aren't they paired because I'm studying the same stores before and after? For example, the "Atlanta" market has 15 stores. I'm measuring those 15 stores from pre-launch and comparing the average of the same 15 stores post-launch. $\endgroup$ – ElPresidente May 13 '14 at 2:58
  • $\begingroup$ This makes more sense... make sure to specify in the original question that the unit of measure here as the store within the region, but you want to make a conclusion about the marketing campaign on the entire region. So now, yes the store is the pairing factor. You'll need to decide how to boil down the pre-launch figures into one number to compare the post-launch figures against. $\endgroup$ – Gary Chung May 13 '14 at 5:04
  • $\begingroup$ Got it. But I have a follow up question. So using my example market, I pair up the 15 stores into before/after values and calculate the paired-sample t-test to determine if the market increased or decreased following the launch. However how is store variability accounted for? Say I have two pairs of stores in the region, both of which increased from 10 units sold to 15 units sold. One store however has a 10wk pre-launch stdev of 2 and the other 6. The paired test only looks at deviations between the sample pairs, correct? But it seems not all pairs are equally significant. $\endgroup$ – ElPresidente May 13 '14 at 13:33
  • $\begingroup$ I agree, that's a tough one of pre-launch stdev. You'll need to put this issue into context of your problem to be able to choose a best approach. Perhaps, for instance, using adding in a weighting factor to adjust for pre-launch stdev. Other methods are possible, but I think they'll be at the sacrifice of interpretability. I'd be happy to work through this with you over Google Chat. Contact info on my profile. $\endgroup$ – Gary Chung May 14 '14 at 0:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.