I have a question about whether I would be adding bias to an A/B test by updating my prior based on combined A & B data, and then running the A/B test on that prior.

My A/B test is click through rates, so I am using a beta-binomial distribution set-up.

I have estimated priors for my population based on previous data, but due to complications it may not accurately represent the population that the test is being run on. As an example, say that my prior is Beta(34, 726) distributed, but if I combine my A and B data (while controlling for sample rates) I can update that prior to Beta(26, 1032).

Would it be valid then to use Beta(26, 1032) as my prior for my A/B test, or am I adding bias by using prior based on my results?

(a) My data is standard binomial distributed data. I am interested in 1 as a success and 0 as a failure. The data is generated by users which I have various characteristics on (age, tenure of user, etc.)

(b) If I did beta-binomial regression on 3 user characteristics, such as the ones I mentioned above, I would get a prior that represents those 3 characteristics. However, due to the nature of the experiment, there are more characteristics that define the population that weren't accounted for in the regression. For example, I know the prior for age 20-28 users. The experiment yields a higher proportion of male users than the total population, and males have different behaviors than females.

(c) I want to update my prior so that it can reflect the different behaviors that weren't accounted for in the regression. I have collected data from a treatment group and a holdout group. I would update the prior using the calculations you listed. I then want to use this prior to estimate the posterior distributions for a treatment group and a holdout group, and calculate the effect of the treatment.

  • $\begingroup$ Could you describe in greater detail: (a) your data, (b) the "complications" you mention, (c) your exact procedure (e.g. how did you update your prior?). Your question is unclear since in standard beta-binomial setting it is impossible for updated prior to ave any of the parameters smaller then previously since it is Beta(alpha+number of successes, beta + number of failures), where since we are talking about counts, they can't be negative... $\endgroup$ – Tim Oct 23 '17 at 15:44
  • $\begingroup$ Thanks for editing, it's much clearer now. I wonder, this sounds like you basically had two logistic regression models and wanted to combine both models, then why won't you do this directly rather then use them to get priors etc. ? $\endgroup$ – Tim Oct 23 '17 at 19:27

I don’t see any problems with the approach to be honest based on the information you’ve provided. I believe that combining the A/B results to update your prior is a reasonable thing to do

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.