Multiple testing for Bayesian revenue models

Question

Following this paper from VWO I have implemented the following model for revenue in a subscription business:

The revenue generated by user $i$ is given by:

$$ \alpha_i \leftarrow Bernoulli\left(\lambda\right) $$ $$ r_i \leftarrow Expon\left(\theta\right) %$$ $$ v_i \leftarrow \alpha_i r_i $$

where $ \alpha_i \in \{0, 1\}$ is an r.v. representing whether user $i$ bought anythign and $r_i \in \mathrm{R}^+$ is the size of the purchase. Finally $v_i$ is the size of the purchase.

In a given A/B test there $n_A$ users in variation A, and $c_A$ sales.

$$ s_A = \frac{1}{c_a}\sum_{k=1}^{c_A} s_A^K $$

Is the revenue per sale in variation A

We assume a $\mathrm{Beta}\left(a, b\right)$ prior on $\lambda$ and a $\mathrm{Gamma}\left(k, \Theta\right) $ on $\theta$. Then the total posterior on $\left(\lambda, \theta\right) $ is

$$ P\left(\lambda, \theta \mid n, c, s\right) = \mathrm{Beta}\left(a+c, b+n-c\right) \mathrm{Gamma}\left(k+c, \frac{\Theta}{1+\Theta c s}\right) $$

From this posterior you can calculate things like:

$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$

by direct simulation.

However, I am interested in comparing multiple hypotheses. In this case, whether

$$ P\left( \lambda_A > \lambda_B \right) $$

and

$$ P\left( \theta_A > \theta_B\right) $$

and

$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$

Simultaneously.

Andrew Gelman in this paper suggest using a heirarchical regression model to soove the multiple comparisons problem. So would a model where I put a hyperprior over $(\lambda_A, \theta_A)$ and $(\lambda_B, \theta_B)$ be sufficient?

Something like:

The revenue generated by user $i$ in group $j$ is given by:

$$ \alpha_{ij} \leftarrow Bernoulli\left(\lambda_j\right) $$ $$ r_{ij} \leftarrow Expon\left(\theta_j\right) $$ $$ v_{ij} \leftarrow \alpha_{ij} r_{ij} $$ $$ \lambda_j \sim \mathcal{D}\left(\lambda, \sigma_{\lambda}\right) $$ $$ \theta_j \sim \mathcal{D}\left(\theta, \sigma_{\theta}\right) $$

where $ \mathcal{D} $ is some suitable distribution I haven't figured out yet.

Answer 1

Do you really mean an A/B test (so A and B are the only variations)? I'll assume you do, and answer accordingly.

The hierarchical regression modelling approach advocated by Gelman is intended for situations where you have a large number of variations, $A, B, ..., Z$, and wish to compare them pairwise: $P(\lambda_A > \lambda_B), P(\lambda_A > \lambda_C), ..., P(\lambda_Y > \lambda_Z)$. It works by modelling all of the $\lambda$ parameters as samples from a population, and shrinking noisy, too-high or too-low estimates towards the population mean.

It doesn't really make sense here, where you have only two variations, and three tests. I don't think it's sensible to assume that estimate of $\lambda_A - \lambda_B$ has any bearing on the estimate of $\theta_A - \theta_B$, and even if it did, with only three tests you don't have enough information to start estimating the distribution of the parameters.

I think there are three more sensible things you can do here.

Stop worrying

You're running three tests, on different parameters, so multiple comparisons aren't such a big concern here (unlike the situation described above, with a large number of variations).

Adjust post-hoc

If you have good reason to want to be conservative in your decisions, you can always estimate, e.g., $P(\lambda_A > \lambda_B)$ normally, and then adjust your estimate to be more conservative. A natural way to do this is by multiplying the odds: $\frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)}_{\text{Adjusted}} = \frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)} \times \frac{1}{3}$ if $P(\lambda_A > \lambda_B)$, and $\times \frac{3}{1}$ otherwise.

(Please check I have the signs right here!)

Use conservative priors

Uninformative, high-variance priors indicate that $A$ and $B$ can vary widely, so are likely to be dissimilar, increasing the chance of finding a large posterior difference. Informative, low-variance priors strongly constrain the values of $\lambda$, reducing the difference between them. The variance of the Beta prior depends on $a+b$. I can't remember offhand what dictates the variance of the Gamma.