The best way for me to explain the problem might be to just give an instance of our data and the type of test/conclusion I am attempting to draw. Unfortunately as the title suggests, my understanding of econometric and statistics methods has waned.

The data consists of transactions, which occur at a restaurant. I am testing the impact of a change on a menu, removing two size options (small and large) and replacing them with one size option (call it medium.) I have about 20 test restaurants, and about 200 control restaurants, so about 10 of each control are mapped to a test for similarity in geography, etc.

That seems straight forward, however, unbeknownst to me, a test had begun to run at the same time which introduced a new flavor of a food item at our test restaurants. So, our restaurants for our testing period had both a size replacement (remove small large, introduce medium) and a flavor introduction (add flavor.) The data we collected includes a control period prior to our test with all restaurants, then a testing period, then a post testing period (which is where we noticed that the introduction of a new flavor may have corrupted our sample, having remembered that to have been the case.)

Is this something that could be solved with a 2 way ANOVA? If so where would I begin to dive in there? What other methods should I be exploring and how should I be setting these up?

I wanted this to be more of a discussion than a simple question and answer (because I doubt I would learn much from that.) But both are fine.

Thanks, Matt


If all the test restaurants had both the size change and flavor change at the same time, then, as described, they are completely confounded and cannot be partitioned. But, because the treatments are quite different in their expected effect (size will affect the purchasing decision instantly, whereas flavor will only affect the next round?), it might be possible to justify testing for the immediate effect before-after control-impact of testing period vs. pre-testing, or, rather, argue that the results of this confounded comparison are due mostly to size, depending on the length of the testing period.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.