Say FB is testing huge, horribly intrusive ads for retirement homes on 90% of its users (call them group A). This angers most people in group A besides 80+ year old users. As a result, most of group A leaves FB permanently. Of course, FB users communicate with each other, so when most of group A leaves, many group B users leave as well.
FB is ultimately left with 80+ year olds who are seeing retirement home ads and some 80+ year olds who do not see them. So, the people in group A who see the ads generate 150% more profit than those in group B. FB might conclude that the ads succeeded, except, the overall profitability of facebook has dropped 90%.
3 questions:
Is this AB test even valid to begin with?
Regardless of your answer to the above, assuming you had to do this test, how would you account for the non-independence of the groups when you interpret the results?
Does the answer to any of the above change if the test is more subtle, yet still may affect both the experimental and control groups? E.g., FB forces 10% of users to have an inbox that flashes bright colors and induces seizures. So some of those people leave and convince a few of their friends in the control group to leave, but the sample sizes are so small, FB's overall profitability does not decrease significantly as a result of the test.