# Multiple testing for Bayesian revenue models

Following this paper from VWO I have implemented the following model for revenue in a subscription business:

The revenue generated by user $$i$$ is given by:

$$\alpha_i \leftarrow Bernoulli\left(\lambda\right)$$ $$r_i \leftarrow Expon\left(\theta\right) %$$ $$v_i \leftarrow \alpha_i r_i$$

where $$\alpha_i \in \{0, 1\}$$ is an r.v. representing whether user $$i$$ bought anythign and $$r_i \in \mathrm{R}^+$$ is the size of the purchase. Finally $$v_i$$ is the size of the purchase.

In a given A/B test there $$n_A$$ users in variation A, and $$c_A$$ sales.

$$s_A = \frac{1}{c_a}\sum_{k=1}^{c_A} s_A^K$$

Is the revenue per sale in variation A

We assume a $$\mathrm{Beta}\left(a, b\right)$$ prior on $$\lambda$$ and a $$\mathrm{Gamma}\left(k, \Theta\right)$$ on $$\theta$$. Then the total posterior on $$\left(\lambda, \theta\right)$$ is

$$P\left(\lambda, \theta \mid n, c, s\right) = \mathrm{Beta}\left(a+c, b+n-c\right) \mathrm{Gamma}\left(k+c, \frac{\Theta}{1+\Theta c s}\right)$$

From this posterior you can calculate things like:

$$P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right)$$

by direct simulation.

However, I am interested in comparing multiple hypotheses. In this case, whether

$$P\left( \lambda_A > \lambda_B \right)$$

and

$$P\left( \theta_A > \theta_B\right)$$

and

$$P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right)$$

Simultaneously.

Andrew Gelman in this paper suggest using a heirarchical regression model to soove the multiple comparisons problem. So would a model where I put a hyperprior over $$(\lambda_A, \theta_A)$$ and $$(\lambda_B, \theta_B)$$ be sufficient?

Something like:

The revenue generated by user $$i$$ in group $$j$$ is given by:

$$\alpha_{ij} \leftarrow Bernoulli\left(\lambda_j\right)$$ $$r_{ij} \leftarrow Expon\left(\theta_j\right)$$ $$v_{ij} \leftarrow \alpha_{ij} r_{ij}$$ $$\lambda_j \sim \mathcal{D}\left(\lambda, \sigma_{\lambda}\right)$$ $$\theta_j \sim \mathcal{D}\left(\theta, \sigma_{\theta}\right)$$

where $$\mathcal{D}$$ is some suitable distribution I haven't figured out yet.

Do you really mean an A/B test (so A and B are the only variations)? I'll assume you do, and answer accordingly.

The hierarchical regression modelling approach advocated by Gelman is intended for situations where you have a large number of variations, $$A, B, ..., Z$$, and wish to compare them pairwise: $$P(\lambda_A > \lambda_B), P(\lambda_A > \lambda_C), ..., P(\lambda_Y > \lambda_Z)$$. It works by modelling all of the $$\lambda$$ parameters as samples from a population, and shrinking noisy, too-high or too-low estimates towards the population mean.

It doesn't really make sense here, where you have only two variations, and three tests. I don't think it's sensible to assume that estimate of $$\lambda_A - \lambda_B$$ has any bearing on the estimate of $$\theta_A - \theta_B$$, and even if it did, with only three tests you don't have enough information to start estimating the distribution of the parameters.

I think there are three more sensible things you can do here.

## Stop worrying

You're running three tests, on different parameters, so multiple comparisons aren't such a big concern here (unlike the situation described above, with a large number of variations).

If you have good reason to want to be conservative in your decisions, you can always estimate, e.g., $$P(\lambda_A > \lambda_B)$$ normally, and then adjust your estimate to be more conservative. A natural way to do this is by multiplying the odds: $$\frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)}_{\text{Adjusted}} = \frac{P(\lambda_A > \lambda_B)}{P(\lambda_A < \lambda_B)} \times \frac{1}{3}$$ if $$P(\lambda_A > \lambda_B)$$, and $$\times \frac{3}{1}$$ otherwise.
Uninformative, high-variance priors indicate that $$A$$ and $$B$$ can vary widely, so are likely to be dissimilar, increasing the chance of finding a large posterior difference. Informative, low-variance priors strongly constrain the values of $$\lambda$$, reducing the difference between them. The variance of the Beta prior depends on $$a+b$$. I can't remember offhand what dictates the variance of the Gamma.
• Ah. In that case, yes, it does make sense to model $\lambda$ and $\theta$ as samples from a population (and it sound like you know how). Modelling $[log(\lambda), log(\theta)]$ as samples from a bivariate Normal distribution would probably be a good place to start. – Eoin Jan 11 at 18:09