Hierarchical model for A/B experiment?

Question

I'm new to Bayesian statistics.

I have a metric that has a very non-parametric distribution, which would make it very difficult to use in an A/B experiment.

However, it can be broken up into multiple levels of distributions like so:

$$ broken = bernoulli(n, p) \\ qualify = bernoulli(n, p) \\ metric \sim lognormal(\mu, \sigma \: | \: broken=0, \: qualify=1) \: + \\ lognormal(\mu, \sigma \: | \: broken=0, \: qualify=0) + \\lognormal(\mu, \sigma \: | \: broken=1) $$

Is this considered a Bayesian Hierarchical Model? And if I ran an A/B experiment, took the test parameters found for those distributions, and simulated the metric through the model, would that be considered a correct / viable method?

Answer 1

If the broken/qualify variables are observable then this sounds appropriate to consider as a hierarchical model; and your process sounds like a reasonable starting point.

However for it to be considered a Bayesian approach you need a few more considerations:

• The Bayesian Hierarchical model assumes that the parameter pairs $(\mu_1,\sigma_1), \, (\mu_2, \sigma_2)$ and $(\mu_3,\sigma_3)$ which correspond to each of the modes you describe (eg. not broken, and qualify) come from some underlying distribution. This is the hyperprior, and you would need to define it explicitly (even if defined as being non-informative).

• If you sample $Y$ directly using the central parameter estimates then you are not going to be making use of any of the uncertainty in your parameters, which is captured by prior distributions you should specify (including the hyperprior referenced above). The Bayesian approach would be to generate posterior distributions for your (hyper)parameters, sample parameters from these distributions, and then sample $Y$ based on those parameter choices.

The above description is assuming that you can directly observe the broken/qualify values, and that estimating their frequency is not of particular interest.

If this is not the case you may wish to add an extra step which is to consider this as a mixture-model. That is we suppose the form

$$I \sim P_3,$$ $$Y = \mathcal{L}(\mu_I, \sigma_I), $$

where $P_3$ is some unknown distribution on a set of three indices $\{1,2,3\}$, which describes the mixture probabilities, and $\mathcal L$ denotes a log-normal distribution.

In addition to describing a hyperprior for the parameters $(\mu_i,\sigma_i)$ you will want a prior on $P_3$. A common starting point is to choose a Dirichlet prior as it is the canonical parametric distribution on a countable set; an alternative is the Logistic normal distribution.

The non-informative Dirichlet hyperprior is $\text{Dir}(1/2,1/2,1/2)$, whose mode would be that each of broken / broken - not qualified / broken - qualified is believed a priori to be equally likely.

Chances are that you have prior knowledge of the situation that you can substantially improve this choice; eg. you may well have views on whether breaks are substantially less frequent than non-breaks, and likewise the frequency of qualify/not qualify.