I am trying to refine the way my company validates tests in retail stores for products that we sell. The prior way was only to look at immediate change in dollar and unit sales without taking into account what would have happened without the test.
The direction I am headed is A/B testing with holdouts that have similar directional sales over the last 52 weeks. There would be a post and pre period to compare the test vs control. My biggest roadblock is trying to convince the team and myself that a Pearson correlation time series on sales is not the correct way to identify a set of control stores for the test.
We want to test anywhere from 50 to 2000 test stores at a time and try and obtain a 1:3 test to control store ratio with offset test dates. I really think we should cluster the stores using time series and choosing the stores with the closest means but am having a hard time convincing others. We would be either using R or even Aster R.
I have also used the R package CausalImpact which seems to be a work around for now but for ramping up to a large scale continuous production testing environment am not sure that will suffice.
In summary what is the best way to identify control stores to match test stores on data sets that can be in excess of 3 million records?