When analyzing experiments with binary action (for example, will a user convert or not), then Beta-Binomial distribution is apt for that.

For example, to model Click Through Rate (CTR):

Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)

Now, Pr(CTR), our prior, can be modeled via Beta distribution with alpha as number of users who clicked, and bete is number who did not click in the variant. Pr(Data | CTR), our likelihood, says what is the chance of getting N clicks given a certain CTR, thus, it is a Binomial distribution.

But I am not able to figure out what distributions to use for the prior and the likelihood for the following cases:

  1. Measuring difference in revenue between A & B. In such case, what we measure is a positive real number instead of a binary one.

  2. Having groups of users as unit of randomization, i.e. some groups will fall in A, and some groups will fall in B. In such case, what distributions to use to model/measure the difference in conversions rates between A & B.

Further Elaboration on Case 2

I think I need to explain case 2 in more details. Say you have 10 groups in the A variant, and 10 in the B variant. Groups can have conversion rates of anything between 0 and 1, for example 0.6 (60% of the users in the group converted). Now we need to calculate the following:

Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)

What is the distribution for Pr(CTR)? Can I use Beta distribution as well? What are the values of alpha and beta here? Remember, we have 10 groups each with CTR between 0 and 1. And, I do not think we can actually use the total number of converted vs non-converted in the whole variant, since our unit of randomization is the group level not the individual level. Is there such as thing as aggregate distribution of Beta's; Bernoulli =(aggregated)=> Binomial, Beta =(aggregated)=> what?

Similarly, what is the distribution for Pr(Data | CTR)

  • $\begingroup$ The difference in revenue is just a real number, isn't it? $\endgroup$ Oct 22 '19 at 21:23
  • $\begingroup$ Yes, mostly yes, but otherwise I can also normalize it by something to make it also between 0-1 if needed $\endgroup$
    – Tarek Amr
    Oct 23 '19 at 12:35
  • $\begingroup$ I was thinking it's probably easier without the non-negativity constraint, actually. Normal distribution would not a be a crazy starting point, even though it can, in theory, predict differences that are larger than either revenue alone. $\endgroup$ Oct 23 '19 at 15:05

The answer to this question depends on a lot of stuff, but some suggestions are:

  1. For modeling the positive real numbers (e.g., revenue), it is usually recommended to use the log-normal distribution (or just take the log of your data and then model it with a normal distribution). As far as prior choice, that's an interesting question and you have a lot of choices. If you want to have something simple, you can go with the conjugate prior distribution: https://en.wikipedia.org/wiki/Normal-inverse-gamma_distribution. Or you can directly place informative priors on the parameters from another distribution (e.g., use a normal distribution to place a prior on the mean of the log of your data and a half normal for the standard deviation; remember standard deviation can't be negative).

Some resources that describe the answer I gave you: Choosing Prior for $\sigma^2$ in the Normal (Polynomial) Regression Model $Y_i | \mu, \sigma^2 \sim \mathcal{N}(\mu_i, \sigma^2)$ and Priors for log-normal models

  1. If you have conversion percentages for groups of users allocated to A or B, then you could model those directly by using a Beta distribution. This is a great use case for the Beta distribution, because it is constrained to be between 0 and 1 and its parameters have a natural interpretation in terms of successes and failures. Assuming you have some prior $\alpha_0$ and $\beta_0$, then the posterior distribution is just: Beta($\alpha_0$ + successes, $\beta_0$ + failutes). So if you have a group of 100 users allocated to A and 40 convert you'd have Beta($\alpha_0$ + 40, $\beta_0$ + 60) as your posterior distribution. This also makes updating the distributions as you get more data quite easy. See here: http://varianceexplained.org/statistics/beta_distribution_and_baseball/
  • $\begingroup$ For the 2nd case; I am not sure if I am allowed to use the total number of success and failure in the variant like this? My issues here, since we randomize the experiment on a group level, shouldn't the analysis be on the group level too? Not total number of successes/failures, but some aggregate of the percentages of successes in each group within the variant or so? Right? $\endgroup$
    – Tarek Amr
    Oct 23 '19 at 13:26
  • $\begingroup$ So for each group you have the conversion rate right? The parametrization I was proposing is the one that I think is easiest to understand in terms of what the parameters mean. But you can of course parametrize it in other ways such as using the mean and variance of your data for each group: stats.stackexchange.com/questions/12232/… $\endgroup$
    – Patrick
    Oct 23 '19 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.