0
$\begingroup$

I have a question about whether I would be adding bias to an A/B test by updating my prior based on combined A & B data, and then running the A/B test on that prior.

My A/B test is click through rates, so I am using a beta-binomial distribution set-up.

I have estimated priors for my population based on previous data, but due to complications it may not accurately represent the population that the test is being run on. As an example, say that my prior is Beta(34, 726) distributed, but if I combine my A and B data (while controlling for sample rates) I can update that prior to Beta(26, 1032).

Would it be valid then to use Beta(26, 1032) as my prior for my A/B test, or am I adding bias by using prior based on my results?

(a) My data is standard binomial distributed data. I am interested in 1 as a success and 0 as a failure. The data is generated by users which I have various characteristics on (age, tenure of user, etc.)

(b) If I did beta-binomial regression on 3 user characteristics, such as the ones I mentioned above, I would get a prior that represents those 3 characteristics. However, due to the nature of the experiment, there are more characteristics that define the population that weren't accounted for in the regression. For example, I know the prior for age 20-28 users. The experiment yields a higher proportion of male users than the total population, and males have different behaviors than females.

(c) I want to update my prior so that it can reflect the different behaviors that weren't accounted for in the regression. I have collected data from a treatment group and a holdout group. I would update the prior using the calculations you listed. I then want to use this prior to estimate the posterior distributions for a treatment group and a holdout group, and calculate the effect of the treatment.

$\endgroup$
2
  • $\begingroup$ Could you describe in greater detail: (a) your data, (b) the "complications" you mention, (c) your exact procedure (e.g. how did you update your prior?). Your question is unclear since in standard beta-binomial setting it is impossible for updated prior to ave any of the parameters smaller then previously since it is Beta(alpha+number of successes, beta + number of failures), where since we are talking about counts, they can't be negative... $\endgroup$
    – Tim
    Oct 23, 2017 at 15:44
  • $\begingroup$ Thanks for editing, it's much clearer now. I wonder, this sounds like you basically had two logistic regression models and wanted to combine both models, then why won't you do this directly rather then use them to get priors etc. ? $\endgroup$
    – Tim
    Oct 23, 2017 at 19:27

2 Answers 2

0
$\begingroup$

I don’t see any problems with the approach to be honest based on the information you’ve provided. I believe that combining the A/B results to update your prior is a reasonable thing to do

$\endgroup$
0
$\begingroup$

If I understand correctly, you are A/B testing, i.e. collecting data under condition A and under B, and you want to know whether the a binomial proportion is higher under condition A compared with B. Additionally, you have a model that gives you possibly good, possibly questionable predictions about what you expect in condition A and B based on some other data that was not part of the A/B test you are about to analyze.

In that case, this is closely related to proof of concept trials in drug development, where one often tries to bring in data on control group outcomes. Here, it seems we have information about both A and B (not just one of them), but that doesn't change all that much. In that setting, approaches like the robust meta-analytic predictive prior, borrowing through hierarchical models and various related alternatives are very popular.

Doing this all without introducing bias/type I error rate inflation is of course challenging and not entirely possible, but these methods aim to minimize the impact of clearly having gotten the prior wrong.

If you have a prior and some new data, you can implement the idea of the second paper above quite easily. E.g. here's one way to do something like that using the brms R package:

library(tidyverse)
library(brms)

# Example data, where the data strongly suggest our prior information on
# B is wrong, while what we observe for A is reasonably compatible with our prior information
example = tibble(group=c("A", "A", "B", "B"), 
                 historicalA=c(1L, 0L, 0L, 0L), # Historical A = pseudo-data for A equivalent to our prior information on condition A
                 historicalB=c(0L, 0L, 1L, 0L), # Historical B = pseudo-data for B equivalent to our prior information on condition B
                 success=c(25, 5, 500, 5), 
                 cases = c(1000, 200, 1000, 200))

brmfit1 = brm( success | trials(cases) ~ 0 + group + historicalA + historicalB,
               family=binomial(link=logit),
               prior = prior(class="b", coef="groupA", normal(0, 3.14)) +
                 prior(class="b", coef="groupB", normal(0, 3.14)) +
                 prior(class="b", coef="historicalA", double_exponential(0, 0.5)) +
                 prior(class="b", coef="historicalB", double_exponential(0, 0.5)),
               data=example)

summary(brmfit1)
```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.