Bayesian A/B test analysis for non-binary actions When analyzing experiments with binary action (for example, will a user convert or not), then Beta-Binomial distribution is apt for that. 
For example, to model Click Through Rate (CTR):
Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)

Now, Pr(CTR), our prior, can be modeled via Beta distribution with alpha as number of users who clicked, and bete is number who did not click in the variant. Pr(Data | CTR), our likelihood, says what is the chance of getting N clicks given a certain CTR, thus, it is a Binomial distribution. 
But I am not able to figure out what distributions to use for the prior and the likelihood for the following cases:


*

*Measuring difference in revenue between A & B. In such case, what we measure is a positive real number instead of a binary one. 

*Having groups of users as unit of randomization, i.e. some groups will fall in A, and some groups will fall in B. In such case, what distributions to use to model/measure the difference in conversions rates between A & B. 

Further Elaboration on Case 2
I think I need to explain case 2 in more details. Say you have 10 groups in the A variant, and 10 in the B variant. Groups can have conversion rates of anything between 0 and 1, for example 0.6 (60% of the users in the group converted). Now we need to calculate the following: 
Pr(CTR | Data) = Pr(Data | CTR) * Pr(CTR)

What is the distribution for Pr(CTR)? Can I use Beta distribution as well? What are the values of alpha and beta here? Remember, we have 10 groups each with CTR between 0 and 1. And, I do not think we can actually use the total number of converted vs non-converted in the whole variant, since our unit of randomization is the group level not the individual level. Is there such as thing as aggregate distribution of Beta's; Bernoulli =(aggregated)=> Binomial, Beta =(aggregated)=> what?  
Similarly, what is the distribution for Pr(Data | CTR)  
 A: The answer to this question depends on a lot of stuff, but some suggestions are:

*

*For modeling the positive real numbers (e.g., revenue), it is usually recommended to use the log-normal distribution (or just take the log of your data and then model it with a normal distribution). As far as prior choice, that's an interesting question and you have a lot of choices. If you want to have something simple, you can go with the conjugate prior distribution: https://en.wikipedia.org/wiki/Normal-inverse-gamma_distribution. Or you can directly place informative priors on the parameters from another distribution (e.g., use a normal distribution to place a prior on the mean of the log of your data and a half normal for the standard deviation; remember standard deviation can't be negative).

Some resources that describe the answer I gave you: Choosing Prior for $\sigma^2$ in the Normal (Polynomial) Regression Model $Y_i | \mu, \sigma^2 \sim \mathcal{N}(\mu_i, \sigma^2)$ and Priors for log-normal models


*If you have conversion percentages for groups of users allocated to A or B, then you could model those directly by using a Beta distribution. This is a great use case for the Beta distribution, because it is constrained to be between 0 and 1 and its parameters have a natural interpretation in terms of successes and failures. Assuming you have some prior $\alpha_0$ and $\beta_0$, then the posterior distribution is just: Beta($\alpha_0$ + successes, $\beta_0$ + failutes). So if you have a group of 100 users allocated to A and 40 convert you'd have Beta($\alpha_0$ + 40, $\beta_0$ + 60) as your posterior distribution. This also makes updating the distributions as you get more data quite easy. See here: http://varianceexplained.org/statistics/beta_distribution_and_baseball/
A: There isn't an accepted answer to this post yet so I will throw in my two cents.

*

*Since revenue is supported on the positive reals, you're going to need a likelihood which respects this property.  The gamma likelihood is often used in financial and insurance applications, making it a good initial model.  The problem is that revenue (especially revenue) can be long tailed, making the gamma potentially inappropriate. This model mispecification may not matter if the effect of the variant is sufficiently large, because after all you're not interested in making probabilistic statements about the revenue, you just want to know if there are differences in the expected revenue.  The gamma distirbution also has a family of conjugate priors you could potentially use, however I think there may be more suitable means of analysis which may or may not be Bayesian.


*Generally this is moving towards regression territory.  If the groups are defined by pre-exposure characteristics, it might be beneficial to use those to explain away variation in the outcome.  Depending on what you're adjusting for, you could simply do a gamma regression (or use some other likelihood) and use a binary indicator for exposure.  However, we are veering away from conjugate prior territory, and this means you'd likely need to use something like Stan, PyMC, or similar to perform the analysis.
