1
$\begingroup$

I'm creating an AB testing framework using Bayesian methods. It's a conversion based test, so users land on the site, randomly get assigned one of two experiences (i.e. group A or group B) and then potentially convert. If I run this test, say, every hour, I'll get a number of people who land and convert within that hour. Then I can easily compare which of the two groups converted at higher rates.

Some people, however, may take 2 hours to convert. Some may take 2 days. I want my model to take into account the fact that one group may convert at a longer time after landing, than the other group.

Does anyone know of a smart way to account for the time-to-conversion component? I'm thinking of comparing conversion rates among cohorts rather than based on time, but after googling around for how people have approached this type of problem, I haven't read anything about it. Surely people aren't ignoring this aspect of their tests.

$\endgroup$
1
$\begingroup$

In general, you don't watch the test as it is performed. It's not that people ignore it but constantly testing leads to an increase in type I errors. If you repeatedly test you need adjust for multiple testing.

If you're doing this in a bayesian framework (like with BEST) then the strength of your priors and the uncertainty in your effect size measurement will help deal with the "delay to conversion" problem.

To be safe, you should just set a time frame before which you won't calculate results. In the NHST paradigm this is usually decided by your sample size calculation. That'd be a good place to estimate this time window even in your bayesian case.

$\endgroup$
1
  • $\begingroup$ Multiple testing is a no-no in the NHST paradigm for the reason you indicated. With Bayesian, however, I can assume that whatever the posterior tells me about the test at the time that I peek is the correct answer, given my data and priors. So I'm not sure why "peeking" is bad in this case. Are you suggesting that I use NHST methods to calculate a sample size ahead of time and then run the Bayesian test? Seems like I'd be conflating two potentially different approaches no? $\endgroup$ – ilanman May 4 '15 at 16:30
1
$\begingroup$

+1 to @cwhardland's answer, and I want to suggest another idea. One way to solve it is to use survival analysis, specifically survival function estimates, to measure how long it took the two groups to convert (where conversion is the death event). For example, your end result might look like:

random experiment

From this, you can see which group converts faster / more often.

$\endgroup$
3
  • 1
    $\begingroup$ Interesting! So would you suggest fitting a survival curve to each cohort that comes in and seeing which group consistently had earlier times to conversion? $\endgroup$ – ilanman May 4 '15 at 16:34
  • $\begingroup$ What do you mean when you say cohort? Just to be clear. $\endgroup$ – Cam.Davidson.Pilon May 5 '15 at 0:42
  • $\begingroup$ When I say cohort, I mean group of people who land in a given hour of the test. We would follow this group over time and see how many of them converted, expecting a majority to do so right away with some trickling in down the road. We do this for every cohort until stopping the test. My main concern is around incorporating: 1) Number of people in a cohort who have already converted 2) Number of people in a cohort who have not already converted (at the time I run the analysis) but who are expected to convert eventually. And do this for every cohort since the start of the test. $\endgroup$ – ilanman May 5 '15 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.