Multiple testing for Bayesian revenue models Following this paper from VWO I have implemented the following model for revenue in a subscription business:
The revenue generated by user $i$ is given by:
$$ \alpha_i \leftarrow Bernoulli\left(\lambda\right) $$
$$ r_i \leftarrow Expon\left(\theta\right) %$$
$$ v_i \leftarrow \alpha_i r_i $$
where $ \alpha_i \in \{0, 1\}$ is an r.v. representing whether user $i$ bought anythign and $r_i \in \mathrm{R}^+$ is the size of the purchase. Finally $v_i$ is the size of the purchase.
In a given A/B test there $n_A$ users in variation A, and $c_A$ sales.
$$ s_A = \frac{1}{c_a}\sum_{k=1}^{c_A} s_A^K $$
Is the revenue per sale in variation A
We assume a $\mathrm{Beta}\left(a, b\right)$ prior on $\lambda$ and a $\mathrm{Gamma}\left(k, \Theta\right) $ on $\theta$. Then the total posterior on $\left(\lambda, \theta\right) $ is
$$ P\left(\lambda, \theta \mid n, c, s\right) = \mathrm{Beta}\left(a+c, b+n-c\right) \mathrm{Gamma}\left(k+c, \frac{\Theta}{1+\Theta c s}\right) $$
From this posterior you can calculate things like:
$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$
by direct simulation.
However, I am interested in comparing multiple hypotheses. In this case, whether
$$ P\left( \lambda_A > \lambda_B \right) $$
and
$$ P\left( \theta_A > \theta_B\right) $$
and
$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$
Simultaneously.
Andrew Gelman in this paper suggest using a heirarchical regression model to soove the multiple comparisons problem. So would a model where I put a hyperprior over $(\lambda_A, \theta_A)$ and $(\lambda_B, \theta_B)$ be sufficient?
Something like:
The revenue generated by user $i$ in group $j$ is given by:
$$ \alpha_{ij} \leftarrow Bernoulli\left(\lambda_j\right) $$
$$ r_{ij} \leftarrow Expon\left(\theta_j\right) $$
$$ v_{ij} \leftarrow \alpha_{ij} r_{ij} $$
$$ \lambda_j \sim \mathcal{D}\left(\lambda, \sigma_{\lambda}\right) $$
$$ \theta_j \sim \mathcal{D}\left(\theta, \sigma_{\theta}\right) $$
where $ \mathcal{D} $ is some suitable distribution I haven't figured out yet.
 A: 
Do you really mean an A/B test (so A and B are the only variations)?
I'll assume you do, and answer accordingly.

The hierarchical regression modelling approach advocated by Gelman
is intended for situations where you have a large number of variations,
$A, B, ..., Z$, and wish to compare them pairwise:
$P(\lambda_A > \lambda_B), P(\lambda_A > \lambda_C), ..., P(\lambda_Y > \lambda_Z)$.
It works by modelling all of the $\lambda$ parameters as samples
from a population, and shrinking noisy, too-high or too-low estimates
towards the population mean.
It doesn't really make sense here, where you have only two variations, and three tests. I don't think it's sensible to assume that estimate of $\lambda_A - \lambda_B$ has any bearing on the estimate of $\theta_A - \theta_B$, and even if it did, with only three tests you don't have enough information to start estimating the distribution of the parameters.
I think there are three more sensible things you can do here.
Stop worrying
You're running three tests, on different parameters, so multiple comparisons aren't such a big concern here (unlike the situation described above, with a large number of variations).
Adjust post-hoc
If you have good reason to want to be conservative in your decisions, you can always estimate, e.g., $P(\lambda_A > \lambda_B)$ normally, and then adjust your estimate to be more conservative.
A natural way to do this is by multiplying the odds: $\frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)}_{\text{Adjusted}} = \frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)} \times \frac{1}{3}$ if $P(\lambda_A > \lambda_B)$, and $\times \frac{3}{1}$ otherwise.
(Please check I have the signs right here!)
Use conservative priors
Uninformative, high-variance priors indicate that
$A$ and $B$ can vary widely, so are likely to be dissimilar,
increasing the chance of finding a large posterior difference.
Informative, low-variance priors strongly constrain the values of $\lambda$,
reducing the difference between them.
The variance of the Beta prior depends on $a+b$. I can't remember offhand
what dictates the variance of the Gamma.
