Bayesian A/B test analysis for non-binary actions

When analyzing experiments with binary action (for example, will a user convert or not), then Beta-Binomial distribution is apt for that.

For example, to model Click Through Rate (CTR):

Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)


Now, Pr(CTR), our prior, can be modeled via Beta distribution with alpha as number of users who clicked, and bete is number who did not click in the variant. Pr(Data | CTR), our likelihood, says what is the chance of getting N clicks given a certain CTR, thus, it is a Binomial distribution.

But I am not able to figure out what distributions to use for the prior and the likelihood for the following cases:

1. Measuring difference in revenue between A & B. In such case, what we measure is a positive real number instead of a binary one.

2. Having groups of users as unit of randomization, i.e. some groups will fall in A, and some groups will fall in B. In such case, what distributions to use to model/measure the difference in conversions rates between A & B.

Further Elaboration on Case 2

I think I need to explain case 2 in more details. Say you have 10 groups in the A variant, and 10 in the B variant. Groups can have conversion rates of anything between 0 and 1, for example 0.6 (60% of the users in the group converted). Now we need to calculate the following:

Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)


What is the distribution for Pr(CTR)? Can I use Beta distribution as well? What are the values of alpha and beta here? Remember, we have 10 groups each with CTR between 0 and 1. And, I do not think we can actually use the total number of converted vs non-converted in the whole variant, since our unit of randomization is the group level not the individual level. Is there such as thing as aggregate distribution of Beta's; Bernoulli =(aggregated)=> Binomial, Beta =(aggregated)=> what?

Similarly, what is the distribution for Pr(Data | CTR)

• The difference in revenue is just a real number, isn't it? Oct 22, 2019 at 21:23
• Yes, mostly yes, but otherwise I can also normalize it by something to make it also between 0-1 if needed Oct 23, 2019 at 12:35
• I was thinking it's probably easier without the non-negativity constraint, actually. Normal distribution would not a be a crazy starting point, even though it can, in theory, predict differences that are larger than either revenue alone. Oct 23, 2019 at 15:05

The answer to this question depends on a lot of stuff, but some suggestions are:

1. For modeling the positive real numbers (e.g., revenue), it is usually recommended to use the log-normal distribution (or just take the log of your data and then model it with a normal distribution). As far as prior choice, that's an interesting question and you have a lot of choices. If you want to have something simple, you can go with the conjugate prior distribution: https://en.wikipedia.org/wiki/Normal-inverse-gamma_distribution. Or you can directly place informative priors on the parameters from another distribution (e.g., use a normal distribution to place a prior on the mean of the log of your data and a half normal for the standard deviation; remember standard deviation can't be negative).

Some resources that describe the answer I gave you: Choosing Prior for $\sigma^2$ in the Normal (Polynomial) Regression Model $Y_i | \mu, \sigma^2 \sim \mathcal{N}(\mu_i, \sigma^2)$ and Priors for log-normal models

1. If you have conversion percentages for groups of users allocated to A or B, then you could model those directly by using a Beta distribution. This is a great use case for the Beta distribution, because it is constrained to be between 0 and 1 and its parameters have a natural interpretation in terms of successes and failures. Assuming you have some prior $$\alpha_0$$ and $$\beta_0$$, then the posterior distribution is just: Beta($$\alpha_0$$ + successes, $$\beta_0$$ + failutes). So if you have a group of 100 users allocated to A and 40 convert you'd have Beta($$\alpha_0$$ + 40, $$\beta_0$$ + 60) as your posterior distribution. This also makes updating the distributions as you get more data quite easy. See here: http://varianceexplained.org/statistics/beta_distribution_and_baseball/
• For the 2nd case; I am not sure if I am allowed to use the total number of success and failure in the variant like this? My issues here, since we randomize the experiment on a group level, shouldn't the analysis be on the group level too? Not total number of successes/failures, but some aggregate of the percentages of successes in each group within the variant or so? Right? Oct 23, 2019 at 13:26
• So for each group you have the conversion rate right? The parametrization I was proposing is the one that I think is easiest to understand in terms of what the parameters mean. But you can of course parametrize it in other ways such as using the mean and variance of your data for each group: stats.stackexchange.com/questions/12232/… Oct 23, 2019 at 13:45

There isn't an accepted answer to this post yet so I will throw in my two cents.

1. Since revenue is supported on the positive reals, you're going to need a likelihood which respects this property. The gamma likelihood is often used in financial and insurance applications, making it a good initial model. The problem is that revenue (especially revenue) can be long tailed, making the gamma potentially inappropriate. This model mispecification may not matter if the effect of the variant is sufficiently large, because after all you're not interested in making probabilistic statements about the revenue, you just want to know if there are differences in the expected revenue. The gamma distirbution also has a family of conjugate priors you could potentially use, however I think there may be more suitable means of analysis which may or may not be Bayesian.

2. Generally this is moving towards regression territory. If the groups are defined by pre-exposure characteristics, it might be beneficial to use those to explain away variation in the outcome. Depending on what you're adjusting for, you could simply do a gamma regression (or use some other likelihood) and use a binary indicator for exposure. However, we are veering away from conjugate prior territory, and this means you'd likely need to use something like Stan, PyMC, or similar to perform the analysis.