Flipping test and control groups

Question

Let's say you're running an A/B test on a website. Let's also say the standard methodology of randomly displaying one version or the other is unusable due to SEO hand-waving. The particular page under test is localized to each city, so the best testing methodology your one stats-educated programmer could come up with is to randomly divide the population of cities into test and control groups.

Now it's a number of weeks later and you have results that seem valid, but that don't satisfy the business people driving the test. They want to switch the test and control groups and re-run the test. This doesn't seem right to your programmer, who would prefer to re-randomize the cities.

Would flipping the test and control groups have any validity at all? Is there any conceivable reason to do that instead of re-randomizing?

Edit to add: Each group contains just over 8000 cities (all in the US).

Answer 1

As I understand it, you randomly assigned cities to treatments A and B and are now going back to the same cities and are considering either

• Assigning any city that previously was in group A to now be in group B and vice versa, or
• Assigning the cities at random to groups A and B again, independent of the initial assignments

The former design is called a crossover design, and would generally be preferred (see the Wikipedia page), particularly as each city serves as its own control in the A/B comparison.

Since you're dealing with the same sets of cities, you'll need to deal with the correlation between results from the same city. The latter design gives groups that saw A/A, B/B, A/B, and B/A. The A/B and B/A groups are most informative, so the crossover design assigns everyone to one of these groups.

Answer 2

One rationalization could be the following: Suppose that people in the treatment cities just became more likely to click on anything relative to people in the control cities. Then, when you switch the assignments, if the treatment truly has no effect, you'd see higher clicks in the treatment-now-control cities relative to the others even though they receive the control in the new round of the experiment. This would suggest that these cities changed and it wasn't the treatment that drove the change.

But, if you randomly chose the cities to begin with, it is highly unlikely that all those cities became more likely to click (or, alternatively, all the control cities became less likely to click).

This, of course, doesn't consider potential long-term impacts of treatment that might linger and contaminate your new results.

A better idea, I'd say, is to repeat the experiment on a whole new set of cities. I don't know whether this is feasible in your situation.

Answer 3

I can think of two reasons to do what they are asking.

1. This indicates a certain lack of faith in randomization, despite 8000 cities involved. So, if you do this reversal (rather than re-randomize) you will get results they believe in and are more likely to act on. And isn't that the point of doing the research? A slightly weaker design that the client has faith in has quite a bit to recommend it.

2. There may be an implicit theory of diminishing returns to advertising here. Let's suppose the response to 1 unit of advertising is X, and the response to the second unit of advertising is .75X. Then with their design, everybody ends up getting 1 unit. If you re-randomize, 25% of cities get 0 units, 50% get 1 unit, and 25% get 2 units. So, the net impact of marketing dollars is smaller. Under this scenario, their scheme means more efficiency to the marketing expenditure, and efficiency has something to recommend it.

BUT WAIT, I'M NOT DONE YET: Point #2 is also an advantage for what you propose, which is re-randomizing. Effectively you get to test the effect of 2 units. There may be diminishing returns, or there may not be. Some older theories of advertising suggest that at the low end there are increasing returns to advertising (e.g. doesn't have an effect unless you see it 3 times). Theories of advertising effects gets us far afield from StackExchange, so I'll stop now with the thought that actual data on the client's situation is a heck of a lot better than textbook theory. This argument may help you win the day.