How would Facebook measure an AB test whose experimental condition affects the control and experimental groups?

Question

Say FB is testing huge, horribly intrusive ads for retirement homes on 90% of its users (call them group A). This angers most people in group A besides 80+ year old users. As a result, most of group A leaves FB permanently. Of course, FB users communicate with each other, so when most of group A leaves, many group B users leave as well.

FB is ultimately left with 80+ year olds who are seeing retirement home ads and some 80+ year olds who do not see them. So, the people in group A who see the ads generate 150% more profit than those in group B. FB might conclude that the ads succeeded, except, the overall profitability of facebook has dropped 90%.

3 questions:

Is this AB test even valid to begin with?

Regardless of your answer to the above, assuming you had to do this test, how would you account for the non-independence of the groups when you interpret the results?

Does the answer to any of the above change if the test is more subtle, yet still may affect both the experimental and control groups? E.g., FB forces 10% of users to have an inbox that flashes bright colors and induces seizures. So some of those people leave and convince a few of their friends in the control group to leave, but the sample sizes are so small, FB's overall profitability does not decrease significantly as a result of the test.

Answer 1

This is a very interesting question, but one without an entirely satisfactory answer. The speculative content certainly increases as you go down.

One problem with randomized experiments is that they are not valid in the presence of what economists call "general equilibrium effects," which are interactions among individuals induced by the treatment being studied. The statisticians named this problem Stable Unit Treatment Value Assumption (SUTVA) violation. It has been recognized at least since Heckman and Smith's 1993 JEP paper, and probably even earlier in the context of other large-scale social experiments like the Job Training Partnership Act or the literature on unionization in the 60s. Without additional structure, experimental evidence simply cannot tell you what the true effect would be. That is the answer to question number one.

How might one patch up the experimental data? One solution is to make explicit, strong assumptions about how the world works. Heckman, Lochner and Taber's paper on tuition policy is a nice example of this approach. My casual empiricism suggests that most firms do not have the patience for this sort of thing.

So how might a firm like FB handle evaluation issues like this? This solution is a bit of a cop-out: limit the size of the treatment group to far below 80%, maybe combined with clever sampling of social clusters (rather than individuals). Both of these minimizes the GE spillovers, and the effect of a full roll-out would presumably be larger for two-sided platform reasons you mention. If the preliminary partial equilibrium numbers look bad, the feature never sees the light of day outside the test. If the results are not significant, you just let the experiment run a bit longer. This doesn't give you the long-run effect, only a lower bound on it. But that might be sufficient to make the right business decision.

There's also a small literature on binary regressor mis-classification error. The experimental estimate is biased towards zero since some control group members are actually "treated", but there are some ways to fix it using an instrumental variables since the probability of mis-classification depends on network membership. I don't know of any existing literature that tries exactly this.

Another way to deal with this is that the spillover actually takes places on FB. If a treated member announces the reasons for his abandonment of the platform in his status, FB knows exactly who saw it. Some other statistical adjustment may be possible if those folks follow.