I'm interested in changing my A/B tests to Bayesian A/B tests, since I recently read several interesting articles and papers on the subject. In particular I would like to apply the approach given in the paper 'Bayesian A/B Testing at VWO' since I think that the expected loss concept is exactly what I have been looking for as a criterion for stopping the test. I will be trying this for conversion.

But, I also need to compare the amount my customers in each group spend. Since this data is not normally distributed, in the past I have used a permutation test for this, giving a p-value; I conclude that the groups spend different amounts if this is below 0.05.

Can I do this in a Bayesian way as well? Ideally I would like to find a method which allows me (as with conversion) to keep track of the expected loss for both groups, and stop the test when one group has an expected loss lower than the acceptable threshold (say 0.1%).

Any suggestions?

  • $\begingroup$ Bayesian analysis generally allows for a much greater amount of flexibility than traditional null hypothesis significance testing. It's not difficult to account for non normality; however, It's hard to give specific advice without knowing what your data looks like. $\endgroup$ Sep 1, 2016 at 22:59
  • 1
    $\begingroup$ From a quick look at this paper, it doesn't seem very good. For example, the author seems to think that significance testing is the only frequentist method, and he presents a particular (and idiosyncratic) loss function as the loss function. $\endgroup$ Sep 3, 2016 at 18:13
  • $\begingroup$ @C.R. Peterson: The data has 3 peaks, each one a curve with a very heavy positive skew; this is due to 3 spend incentive thresholds. I was wondering if some sort of mixture model was therefore appropriate, perhaps of gamma distributions or skew-normals. $\endgroup$
    – EBartrum
    Sep 5, 2016 at 10:21

2 Answers 2


Let's take a look at what information you have:

  1. Whether the customer purchased anything $(Y_K)$. This is binary.
  2. If the customer made a purchase, which incentive threshold the they were willing to buy at $(Y_L)$. This is ordinal, and ranges from 0 to 3. The value of this is meaningless when $Y_K=0$.
  3. The amount of money the customer spent ($Y_M$). This is a continuous, positive value, and does not include customers for whom $Y_K=0$.
  4. The test treatment $(X)$, which could plausibly affect the above $Y$ values.

I's often helpful to try to model the process that generated the data you're analyzing. The amount of money a person spends $(Y_M)$ appears to be influenced by the incentive they desire ($Y_L$), which is contingent on whether they decide to purchase anything at all ($Y_K$). This suggests something like the following model:

$$\begin{align} Y_{K} & \sim\text{Bernoulli}(p_{x})\\ Y_{L} & \sim\text{Ordered-Logistic}(\eta_{x},c)\\ Y_{M} & \sim\text{LogNormal}(\alpha_{L}+\beta_{x}, \sigma)\end{align}$$

$p_x$ is probability of making a purchase for group $X$. You can either put a prior on the $p$ parameters directly or use a logistic regression.

The ordered logistic regression has $\eta_x$ as the predictor for which incentive group the customer would choose given their A/B group membership. $c$ is a vector of cutpoint parameters that determine what values of $Y_L$ $\eta$ refers over its range. These parameters can be transformed to calculate the posterior probability of a given treatment selecting each incentive level. For more details, I'd recommend reading up on ordered logit regression.

$\alpha_L$ is the intercept for $Y_M$, and it depends on the value of $Y_L$. This should (in theory) take care of the multimodality. Given that you know what the incentive thresholds are, you should use informed priors for this parameter. $\beta_x$ is the effect of one of your treatment groups. I suggested a log-normal distribution since it is easily parameterized and is often used for positive data with a skew, but you should feel free to try other options. You may also wish to consider investigating whether $\beta_x$ or $\sigma$ should vary among levels of $Y_L$.

You should then be able to a loss function to the posterior distribution of the parameters to determine when you should stop the test.

The whole model (including posterior transformations and loss function calculation) could be implemented in Stan.


Take the 99% interval ( HDR - High Density Region ) of your posterior distribution at each step, and check if your Ho ( stopping point ) is in it.

If you dont know the distribution you can simply take a big sample from the posterior and then take the interval based on the percentiles of your sample.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.