I'm running an A/B test. The test is a funnel that, at some step in the funnel, sends half the population to experience A, the other half to experience B. Traditionally everyone saw experience A at this step in the funnel, so I have plenty of historical data for my prior on A. However, I have no data at this step in the funnel for Group B. I do, however, have data on Group B, just at a different step in the funnel.

I expect both groups to convert at similar rates, but the point of the test is to determine which converts higher.

I can argue for using either historical A data as my prior for B, and at the same time argue for using historical B data (which happens at a different step, but behaves similarly as A).

Conceptually, how would I reconcile between these two choices? Other than using my domain knowledge, is there a more scientific way to select the prior?

And yes, I understand that selecting a prior is inherently subjective, but I'd like to minimize that if possible.

  • $\begingroup$ Why not use the combined data from both A an B? Use the historical data from group A from some particular step $c$ and historical data from group B from some other step $d$. $\endgroup$ – missingdataguy Jun 4 '15 at 18:20
  • $\begingroup$ To be sure, that makes sense - however is that approach preferable mathematically speaking to, say, using only data from A or B? This approach kind of "splits the difference" by combining data from each. $\endgroup$ – ilanman Jun 8 '15 at 16:25
  • $\begingroup$ @ilanman it might not be better, but I don't see up front why it would be worse $\endgroup$ – shadowtalker Jun 13 '15 at 22:55

To reconcile between the two, it might help to be explicit about the assumptions inherent in each choice of prior, and then to imagine how you might justify these assumptions to a skeptical audience. For instance, let's say your funnel has only two steps (1) a banner to get someone to a sales page (2) a final sales page that collects e-commerce data and processes a transaction.

Now let's say you use historical data for treatment A as a prior for treatment B. To gauge the strength of this assumption, consider that it's mathematically equivalent to having observed an equal-sized dataset of the B group at the second stage. In a comparison, this seems a problematic assumption to me, even prior hacking. But perhaps your domain knowledge suggests otherwise.

This is one reason you may want to consider a sensitivity analysis using several priors, and consider your audience. For example, if treatment A were an honest description of a boutique cell service, you'd have a hard time convincing yours truly that viewers lured to the sales page with a misleading, "free month of service" ad would convert similarly at the second stage. Unless, of course, you showed me indistinguishable posterior distributions of the conversion at stage 2, both developed from flat priors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.