Bayesian AB testing with time lag

Question

I'm creating an AB testing framework using Bayesian methods. It's a conversion based test, so users land on the site, randomly get assigned one of two experiences (i.e. group A or group B) and then potentially convert. If I run this test, say, every hour, I'll get a number of people who land and convert within that hour. Then I can easily compare which of the two groups converted at higher rates.

Some people, however, may take 2 hours to convert. Some may take 2 days. I want my model to take into account the fact that one group may convert at a longer time after landing, than the other group.

Does anyone know of a smart way to account for the time-to-conversion component? I'm thinking of comparing conversion rates among cohorts rather than based on time, but after googling around for how people have approached this type of problem, I haven't read anything about it. Surely people aren't ignoring this aspect of their tests.

Answer 1

In general, you don't watch the test as it is performed. It's not that people ignore it but constantly testing leads to an increase in type I errors. If you repeatedly test you need adjust for multiple testing.

If you're doing this in a bayesian framework (like with BEST) then the strength of your priors and the uncertainty in your effect size measurement will help deal with the "delay to conversion" problem.

To be safe, you should just set a time frame before which you won't calculate results. In the NHST paradigm this is usually decided by your sample size calculation. That'd be a good place to estimate this time window even in your bayesian case.

Answer 2

+1 to @cwhardland's answer, and I want to suggest another idea. One way to solve it is to use survival analysis, specifically survival function estimates, to measure how long it took the two groups to convert (where conversion is the death event). For example, your end result might look like:

From this, you can see which group converts faster / more often.

Answer 3

A few years late to the party, but some people might still find this useful:

In short, you can define a pdf (or pmf) as follows:

$$P(X=k) = (1-p)\cdot \mathbb{1}_{\{\infty\}}(k) + p\cdot \mathbb{1}_{\geq 0}(k)\cdot Q(k)$$ where $Q$ is a time-to-event-distribution (e.g. the exponential distribution for continuous time or the geometric distribution for discrete time) and $p$ is the conversion probability, i.e. the probability that the user converts at all (at any point in the future). The random variable $X$ define the time-lag to conversion with $X=\infty$ if there is no conversion.

The slightly more tricky part is then the censoring aspect, which however is a standard problem in survival analysis as @Cam.Davidson.Pilon pointed out.

To define a full Bayesian model (e.g. in Stan or PyMC) you need to write out the log-likelihood, which in this case also contains the complementary CDF (aka ccdf) of the variable $X$.

I wrote a blog post precisely about the question you are asking: here that also links to a Stan model I wrote for this.

Hope this is helpful.