Suppose that you are a product manager who wants to run a test for a new product. You consult your statistician, select a sample of customers and run the test. Apparently, another product manager wanted to run a cd testing for his own product, selected a sample and run his own test. The two sample overlapped, so some of the customers were exposed to a and then c, some to c and them to b, etc. It is also possible that some of the customers were also included in tests to some other products. You want to estimate the net effect of one of the test clasdes, say b. How would you handle the situation?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I think the easiest thing is to redefine treatment to be combinations of the two experiments, for a total of four experimental groups (ac, ad, bc, and bd). This assumes that the randomization in the two tests was unrelated.
For the AB test effect, the ad versus bd is the relevant comparison (assuming b and d are the two control variants), but there are several other interesting comparisons that could be made to detect interactions between treatments. It is probably easiest to do this in some sort of regression setting so you can test the coefficients directly.
