I have a question about whether I would be adding bias to an A/B test by updating my prior based on combined A & B data, and then running the A/B test on that prior.
My A/B test is click through rates, so I am using a beta-binomial distribution set-up.
I have estimated priors for my population based on previous data, but due to complications it may not accurately represent the population that the test is being run on. As an example, say that my prior is Beta(34, 726) distributed, but if I combine my A and B data (while controlling for sample rates) I can update that prior to Beta(26, 1032).
Would it be valid then to use Beta(26, 1032) as my prior for my A/B test, or am I adding bias by using prior based on my results?
(a) My data is standard binomial distributed data. I am interested in 1 as a success and 0 as a failure. The data is generated by users which I have various characteristics on (age, tenure of user, etc.)
(b) If I did beta-binomial regression on 3 user characteristics, such as the ones I mentioned above, I would get a prior that represents those 3 characteristics. However, due to the nature of the experiment, there are more characteristics that define the population that weren't accounted for in the regression. For example, I know the prior for age 20-28 users. The experiment yields a higher proportion of male users than the total population, and males have different behaviors than females.
(c) I want to update my prior so that it can reflect the different behaviors that weren't accounted for in the regression. I have collected data from a treatment group and a holdout group. I would update the prior using the calculations you listed. I then want to use this prior to estimate the posterior distributions for a treatment group and a holdout group, and calculate the effect of the treatment.