Interview Question: A/B Survey Results Non-Random Sampling

Question

How would you respond to the following case/question:

Assume you are testing the effect of an ad campaign at a social media company. The study is divided into two groups, 5,000 subjects who are shown an ad for why Coffee Shop X is better than Coffee Shop Y (group A), and 5,000 subjects who are not shown the ad (group B). The subjects are given a survey with a simple yes/no question of "do you find Coffee Shop X better than Coffee Shop Y?". Group A responds yes to the question 70% of the time, while group B responds yes to the question 65% of the time.

However, the samples are not representative of the overall population (assume for standard reasons why surveys are biased). But being a social media company with extensive data, you have access to thousands of features on demographics (i.e., age, sex, etc.) and usage characteristics (i.e., time spent on pages, subjects of interest, etc.) for both the samples and the overall population. What would be your framework for assessing whether the ad campaign really works (convinces more people to prefer Coffee Shop X than in the case of no ad)?

My initial inclination would be that you should build a predictive model with yes/no as the outcome and various features as the predictors on the sample data.

Answer 1

My initial inclination would be that you should build a predictive model with yes/no as the outcome and various features as the predictors on the sample data.

This would be useful in so far as creating a propensity score model, but it is not sufficient in and of itself. The problem is that building a predictive model would only tell you who is likely to say yes/no conditioned on the data you have, which were admittedly biased.

Even though the experiment is designed as a randomized control trial (or AB test), there is the problem of selection bias. Namely, people self select to respond to surveys, and so the effect you estimate is not the effect of the ad campaign, it is the effect of the add campaign on those people who are the type of person to respond to surveys. Very different.

This paper offers some insight into the problem. Let $y$ be the outcome of interest, $x$ be a binary indicator for exposure, and $S$ be a binary indicator for entry into the data pool (i.e. data are collected if $S=1$ and are not otherwise). In the introduction, the authors allude to a corollary they present in which they demonstrate that the conditional distribution $P(y\vert x)$ (which is what we want) is not recoverable if $S$ is a function of both the exposure $x$ and the outcome $y$. So if being prompted about liking coffee shop X better than coffee shop Y and actually liking one shop over the other affect your probability of responding to the survey, then the problem can not be solved; we could not obtain an unbiased estimate of the treatment effect. So, we would need to start with the assumption that this is not true. From there, the linked paper presents some methods for possibly obtaining $P(y\vert x)$ from $P(y\vert x, S=1)$. I would start there.

Answer 2

Because this is an interview question, I'll structure my response in formats that are popular in case study circles:

• First, a clarifying question. Can we trust the survey? That is, was the survey designed and performed with concerns for non-response, biasedness in mind?
  • I'll assume we can trust it and the headline result. This gives us a good starting point for our analyses.
• Next, I'd want to see if the people we care about reaching are in both groups. Who responded to the surveys? Does it make sense to compare these groups, or do we have apples and oranges? To do this, I'll compare summary statistics on some set of observed baseline characteristics I hypothesize are relevant to preferring cafes (demographics, region, etc.).
• If the groups are not too different baseline, I'll proceed with a selection-on-observables strategy and estimate if there's a meaningful difference in response by group. A popular way to do this is by assigning weights to the individuals via propensity score matching and then estimating a regression of the outcome on the treatment indicator (group A or B). The results from this will help me give a rigorous answer to the question. (There's careful work to be done for calculating the right standard error for the treatment estimate, but I'll skip details here in an interview)
• Finally, I'd present results from the survey and my analysis to the stakeholder and have them weigh in on the approach.