Updating Bayesian prior & likelihood for A/B test

Question

I'm fairly new to bayesian. I'm trying to edit a bayesian python code for $A/B$ test analysis. I'm using uninformative priors as a beta distribution, so my $\alpha$ & $\beta$ parameters are $1$ & $1$ for both control & test for the first observation of the data. I have a function which takes in priors, visitors for control, visitors for test and their respective conversions to calculate the posterior

I'm using this Bayesian formula to update the priors -> $\textrm{Beta}(\alpha, \beta)$, and add the successes from the data, $x$, to $\alpha$ and the failures, $n – x$, to $\beta$, and there’s your posterior, $\textrm{Beta}(\alpha+x, \beta+n-x)$. Since this is for an $A/B$ test, i'm using cumulative visitors & conversions for each day as my likelihood and updating the priors from the formula above.

My question is, Should I use cumulative visitors & conversions for each day as my likelihood or visitors & conversions for each day separately since I'm updating the previous days' data in priors?

So my doubt is am I updating previous information in both my likelihood & prior?

Answer 1

Firstly, as pointed out in the answer by @AdamO Bayesian updating results in the same answer, if you add new data in multiple steps or all at once - if they are for the same parameter.

Secondly, I'd wonder whether your observations should be treated as indepdent binary events. Are these not some of the same people, are there not perhaps correlations from day-to-day? Ideally, you'd do a hierarchical model account for e.g. days and whether records refer to the same people. I love a good Bayesian model, but with enough data there might be no difference how you analyze it. If you want be Bayesian, then e.g. R has the great rstanarm package that would let you do this easily, but Python has no excact equivalent, I think, so you could go for a frequentist version in pymer4 (a lot like lme4 in R, apparently). But you may not know what records are from the same people and may not be able to do that. However, in that case be aware that your evidence might be somewhat overstated and that it might be hard to know by how much.

Answer 2

If the goal is to obtain $Pr(\theta | Y_1, Y_2, \ldots, Y_n)$ where the $Y$s are the cumulative data, then a simply application of Bayes Rule shows that:

$$Pr(\theta | Y_1, Y_2, \ldots, Y_n) = \frac{Pr( Y_1, Y_2, \ldots, Y_n | \theta) Pr(\theta)}{Pr(Y_1, Y_2, \ldots, Y_n)}$$

And under the assumption of independence of the $Y$s (being consecutive observations across time):

$$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2) \cdot \ldots \cdot Pr(Y_n)$$

and also

$$Pr(Y_1, Y_2, \ldots, Y_n | \theta) = Pr(Y_1| \theta) \cdot Pr( Y_2| \theta) \cdot \ldots \cdot Pr(Y_n| \theta)$$

So you see that yes you can run the analyses sequentially if the time series is independent.

Alternately, if there is a time series autocorrelated effect, you can rely on the general factorization of the likelihood

$$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2 | Y_1) \cdot \ldots \cdot Pr(Y_n | Y_{n-1} \ldots, Y_2, Y_1)$$

So by simply modeling the autocorrelation using some form of ARIMA or something, you can run a sequential analysis.