# Bayesian A/B test analysis for non-binary actions

When analyzing experiments with binary action (for example, will a user convert or not), then Beta-Binomial distribution is apt for that.

For example, to model Click Through Rate (CTR):

Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)


Now, Pr(CTR), our prior, can be modeled via Beta distribution with alpha as number of users who clicked, and bete is number who did not click in the variant. Pr(Data | CTR), our likelihood, says what is the chance of getting N clicks given a certain CTR, thus, it is a Binomial distribution.

But I am not able to figure out what distributions to use for the prior and the likelihood for the following cases:

1. Measuring difference in revenue between A & B. In such case, what we measure is a positive real number instead of a binary one.

2. Having groups of users as unit of randomization, i.e. some groups will fall in A, and some groups will fall in B. In such case, what distributions to use to model/measure the difference in conversions rates between A & B.

Further Elaboration on Case 2

I think I need to explain case 2 in more details. Say you have 10 groups in the A variant, and 10 in the B variant. Groups can have conversion rates of anything between 0 and 1, for example 0.6 (60% of the users in the group converted). Now we need to calculate the following:

Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)


What is the distribution for Pr(CTR)? Can I use Beta distribution as well? What are the values of alpha and beta here? Remember, we have 10 groups each with CTR between 0 and 1. And, I do not think we can actually use the total number of converted vs non-converted in the whole variant, since our unit of randomization is the group level not the individual level. Is there such as thing as aggregate distribution of Beta's; Bernoulli =(aggregated)=> Binomial, Beta =(aggregated)=> what?

Similarly, what is the distribution for Pr(Data | CTR)

• The difference in revenue is just a real number, isn't it? – Matt Krause Oct 22 '19 at 21:23
• Yes, mostly yes, but otherwise I can also normalize it by something to make it also between 0-1 if needed – Tarek Amr Oct 23 '19 at 12:35
• I was thinking it's probably easier without the non-negativity constraint, actually. Normal distribution would not a be a crazy starting point, even though it can, in theory, predict differences that are larger than either revenue alone. – Matt Krause Oct 23 '19 at 15:05

Some resources that describe the answer I gave you: Choosing Prior for $\sigma^2$ in the Normal (Polynomial) Regression Model $Y_i | \mu, \sigma^2 \sim \mathcal{N}(\mu_i, \sigma^2)$ and Priors for log-normal models
2) If you have conversion percentages for groups of users allocated to A or B then you could model those directly by using a Beta distribution. This is a great use case for the Beta distribution because it is constrained to be between 0 and 1 and its parameters have a natural interpretation in terms of successes and failures. Assuming you have some prior $$\alpha_0$$ and $$\beta_0$$ then the posterior distribution is just: Beta($$\alpha_0$$ + successes, $$\beta_0$$ + failutes). So if you have a group of 100 users allocated to A and 40 convert you'd have Beta($$\alpha_0$$ + 40, $$\beta_0$$ + 60) as your posterior distribution. This also makes updating the distributions as you get more data quite easy. See here: http://varianceexplained.org/statistics/beta_distribution_and_baseball/