I'm running an AB Test and using Survival Analysis to estimate the Return Rate to the website.

Each day (in a total of 7 days) I randomly assign 100 users to group A and 100 users to group B and after 7 days I stop the test.

Each day I also check the number of tracked users (only the ones participating in the experiment) who returned to the website.

At the end of the experiment, there will be:

  • 7 cohorts that passed through day 0
  • 6 cohorts through day 1
  • 5 cohorts through day 2
  • ...
  • 1 cohort through day 6

by Cohort I mean: on the first day of the experiment I assign 100 users to each group and that's my first cohort. On the second day of the experiment I assign another 100 users to each group, and these users are my second cohort, and so on.

As an example, the Return Rate of group A is simply the number of users of group A who returned to the website on day $i$ divided by the number of users that reached that day.

$$ RR_i = \frac{n_i}{N_i}$$

  • $RR_i$ is the Return Rate on day $i$
  • $N_i$ is the number of users that reached day $i$ (if $i = 2$, the number of cohorts that reached that day is 5, if each cohort has 100 users, then $N_2 = 500$)
  • $n_i$ the number of users that returned on that day.

When using Survival Analysis, we assume that when an individual die, of course it never comes back. But in my analysis, an user that comes to the website on day 0, can return on day 1 and 2, or return on day 1, 3 and 6, or even return each of the following 7 days.

In that case, can I still use Survival Analysis?

  1. If so, what else do I need to consider in order to still make it valid?
  2. If not, what better approaches can I use?
  • $\begingroup$ Perhaps providing a quantitative definition of "Return Rate" might help clear things up. If you had a perfect crystal ball that could show you all details of your website usage throughout the future, so that there are no concerns about sampling or uncertainty, exactly how would you compute the Return Rate? $\endgroup$ – whuber Jun 2 '15 at 19:55
  • 2
    $\begingroup$ I would like to suggest that it can be helpful to distinguish terms of importance to survival analysis from terms of importance to you. In order not to get stuck in potentially irrelevant ideas, could we forget about your experiment and survival analysis for a moment? What exactly do you want to learn about your website users? Would it be something like, say, a function $f$ for which $f(i)$ tells you what proportion returns $i$ days after first encountering the site? $\endgroup$ – whuber Jun 2 '15 at 20:24
  • 1
    $\begingroup$ Are you trying to predict how many will return ? if so survival analysis is not the right approach. You might want to assume some type of probability distributions for the following: Some will return, some will never return and there are differences (heterogeneity) among people who are coming to your website and model this as a mixture model. I read an article in European journal of operational research or computers and operations research about a model. If i find it, I'll share it. $\endgroup$ – forecaster Jun 2 '15 at 20:29
  • $\begingroup$ The objetive is to compare the Return Rate of the two groups and decide each one is better day-by-day. The output then would be given as a ratio between $\frac{RR_Bi}{RR_Ai} - 1$ where A is my control group and B the experiment group. I also would want to compute the $SE$ of that ratio. $\endgroup$ – Thiago Balbo Jun 2 '15 at 20:37
  • $\begingroup$ Great @forecaster! It would be great to read that article! Btw, thinking about this problem some days ago I came up with - I guess wrong - solution to this question 'cause I wasn't considering the time. If it's something of interest, here is the question's link. $\endgroup$ – Thiago Balbo Jun 2 '15 at 21:08

If you modeled this with Poisson regression, you would still be able to incorporate the time aspect and not violate the assumptions of the model since the outcome can accommodate multiple hits. You will need to include a strata() term if you have multiple datalines with the same subject ID.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.