Following this paper from VWO I have implemented the following model for revenue in a subscription business:

The revenue generated by user $i$ is given by:

$$ \alpha_i \leftarrow Bernoulli\left(\lambda\right) $$ $$ r_i \leftarrow Expon\left(\theta\right) %$$ $$ v_i \leftarrow \alpha_i r_i $$

where $ \alpha_i \in \{0, 1\}$ is an r.v. representing whether user $i$ bought anythign and $r_i \in \mathrm{R}^+$ is the size of the purchase. Finally $v_i$ is the size of the purchase.

In a given A/B test there $n_A$ users in variation A, and $c_A$ sales.

$$ s_A = \frac{1}{c_a}\sum_{k=1}^{c_A} s_A^K $$

Is the revenue per sale in variation A

We assume a $\mathrm{Beta}\left(a, b\right)$ prior on $\lambda$ and a $\mathrm{Gamma}\left(k, \Theta\right) $ on $\theta$. Then the total posterior on $\left(\lambda, \theta\right) $ is

$$ P\left(\lambda, \theta \mid n, c, s\right) = \mathrm{Beta}\left(a+c, b+n-c\right) \mathrm{Gamma}\left(k+c, \frac{\Theta}{1+\Theta c s}\right) $$

From this posterior you can calculate things like:

$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$

by direct simulation.

However, I am interested in comparing multiple hypotheses. In this case, whether

$$ P\left( \lambda_A > \lambda_B \right) $$


$$ P\left( \theta_A > \theta_B\right) $$


$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$


Andrew Gelman in this paper suggest using a heirarchical regression model to soove the multiple comparisons problem. So would a model where I put a hyperprior over $(\lambda_A, \theta_A)$ and $(\lambda_B, \theta_B)$ be sufficient?

Something like:

The revenue generated by user $i$ in group $j$ is given by:

$$ \alpha_{ij} \leftarrow Bernoulli\left(\lambda_j\right) $$ $$ r_{ij} \leftarrow Expon\left(\theta_j\right) $$ $$ v_{ij} \leftarrow \alpha_{ij} r_{ij} $$ $$ \lambda_j \sim \mathcal{D}\left(\lambda, \sigma_{\lambda}\right) $$ $$ \theta_j \sim \mathcal{D}\left(\theta, \sigma_{\theta}\right) $$

where $ \mathcal{D} $ is some suitable distribution I haven't figured out yet.


1 Answer 1


Do you really mean an A/B test (so A and B are the only variations)? I'll assume you do, and answer accordingly.

The hierarchical regression modelling approach advocated by Gelman is intended for situations where you have a large number of variations, $A, B, ..., Z$, and wish to compare them pairwise: $P(\lambda_A > \lambda_B), P(\lambda_A > \lambda_C), ..., P(\lambda_Y > \lambda_Z)$. It works by modelling all of the $\lambda$ parameters as samples from a population, and shrinking noisy, too-high or too-low estimates towards the population mean.

It doesn't really make sense here, where you have only two variations, and three tests. I don't think it's sensible to assume that estimate of $\lambda_A - \lambda_B$ has any bearing on the estimate of $\theta_A - \theta_B$, and even if it did, with only three tests you don't have enough information to start estimating the distribution of the parameters.

I think there are three more sensible things you can do here.

Stop worrying

You're running three tests, on different parameters, so multiple comparisons aren't such a big concern here (unlike the situation described above, with a large number of variations).

Adjust post-hoc

If you have good reason to want to be conservative in your decisions, you can always estimate, e.g., $P(\lambda_A > \lambda_B)$ normally, and then adjust your estimate to be more conservative. A natural way to do this is by multiplying the odds: $\frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)}_{\text{Adjusted}} = \frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)} \times \frac{1}{3}$ if $P(\lambda_A > \lambda_B)$, and $\times \frac{3}{1}$ otherwise.

(Please check I have the signs right here!)

Use conservative priors

Uninformative, high-variance priors indicate that $A$ and $B$ can vary widely, so are likely to be dissimilar, increasing the chance of finding a large posterior difference. Informative, low-variance priors strongly constrain the values of $\lambda$, reducing the difference between them. The variance of the Beta prior depends on $a+b$. I can't remember offhand what dictates the variance of the Gamma.

  • $\begingroup$ Thanks! I concetrated on the simple 2 variation case here, but in reality there will 3+ variations, and we would like to run comparisons of conversion rate and mean spend against for all variations. $\endgroup$
    – Tom Kealy
    Commented Jan 11, 2021 at 15:39
  • $\begingroup$ Ah. In that case, yes, it does make sense to model $\lambda$ and $\theta$ as samples from a population (and it sound like you know how). Modelling $[log(\lambda), log(\theta)]$ as samples from a bivariate Normal distribution would probably be a good place to start. $\endgroup$
    – Eoin
    Commented Jan 11, 2021 at 18:09
  • $\begingroup$ Out of interest, is it still called "A/B testing" when you have more than two conditions? $\endgroup$
    – Eoin
    Commented Jan 11, 2021 at 18:09
  • $\begingroup$ Yes - in the industry an A/B test can have more than 2 variations. $\endgroup$
    – Tom Kealy
    Commented Mar 10, 2021 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.