A company wishes to test a the effect of a marketing promotion on sales. The historical procedure is "pre-post net of control" (PPNOC). Sales data is taken before a test from a test group and a control group, then sales data is taken after the test. The net of control is then calculated to be the percent increase or decrease of the test group vs the control.

To me it seems that an ANCOVA would be more appropriate or even a mixed model. Are their other alternative that would be better?


Pre-Post net of control is a nice method because it handles seasonality in sales and as long as you pick a representative group and a comparable control group, you should get unbiased estimates. However, one challenge with Pre-Post net of control is you are assuming your data are independent. Sales data usually has auto-correlation, which means you need to account for that or handle it appropriately if you want unbiased standard errors or p-values.

One way to handle this would be to use a mixed model, as you suggested. This would allow you to include random effects such as day, week, and/or month to handle the auto-correlated nature of your data. You could also include other covariates (such as a store ID, demographics, etc), as random effects.

Another way to analyze this (that gets you out of all data assumptions) would be to use a non-parametric method such as simulation (provided you have historical data). For example, you could pick a random time for the test, randomly pick a control and treatment group, and then run 1000 tests using historical data to get a distribution of possible effect sizes due to chance. Then, you could compare the effect size seen in your experiment to the distribution of random effects to see whether your test falls in the 97.5% or 2.5% tails, indicating significance at the 95% confidence level.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.