# Updating Bayesian prior & likelihood for A/B test

I'm fairly new to bayesian. I'm trying to edit a bayesian python code for $A/B$ test analysis. I'm using uninformative priors as a beta distribution, so my $\alpha$ & $\beta$ parameters are $1$ & $1$ for both control & test for the first observation of the data. I have a function which takes in priors, visitors for control, visitors for test and their respective conversions to calculate the posterior

I'm using this Bayesian formula to update the priors -> $\textrm{Beta}(\alpha, \beta)$, and add the successes from the data, $x$, to $\alpha$ and the failures, $n – x$, to $\beta$, and there’s your posterior, $\textrm{Beta}(\alpha+x, \beta+n-x)$. Since this is for an $A/B$ test, i'm using cumulative visitors & conversions for each day as my likelihood and updating the priors from the formula above.

My question is, Should I use cumulative visitors & conversions for each day as my likelihood or visitors & conversions for each day separately since I'm updating the previous days' data in priors?

So my doubt is am I updating previous information in both my likelihood & prior?

• What is the probability to see two independent questions on that topic on X validated within 10 hours?! Jan 15, 2016 at 8:52

Firstly, as pointed out in the answer by @AdamO Bayesian updating results in the same answer, if you add new data in multiple steps or all at once - if they are for the same parameter.

Secondly, I'd wonder whether your observations should be treated as indepdent binary events. Are these not some of the same people, are there not perhaps correlations from day-to-day? Ideally, you'd do a hierarchical model account for e.g. days and whether records refer to the same people. I love a good Bayesian model, but with enough data there might be no difference how you analyze it. If you want be Bayesian, then e.g. R has the great rstanarm package that would let you do this easily, but Python has no excact equivalent, I think, so you could go for a frequentist version in pymer4 (a lot like lme4 in R, apparently). But you may not know what records are from the same people and may not be able to do that. However, in that case be aware that your evidence might be somewhat overstated and that it might be hard to know by how much.

If the goal is to obtain $Pr(\theta | Y_1, Y_2, \ldots, Y_n)$ where the $Y$s are the cumulative data, then a simply application of Bayes Rule shows that:

$$Pr(\theta | Y_1, Y_2, \ldots, Y_n) = \frac{Pr( Y_1, Y_2, \ldots, Y_n | \theta) Pr(\theta)}{Pr(Y_1, Y_2, \ldots, Y_n)}$$

And under the assumption of independence of the $Y$s (being consecutive observations across time):

$$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2) \cdot \ldots \cdot Pr(Y_n)$$

and also

$$Pr(Y_1, Y_2, \ldots, Y_n | \theta) = Pr(Y_1| \theta) \cdot Pr( Y_2| \theta) \cdot \ldots \cdot Pr(Y_n| \theta)$$

So you see that yes you can run the analyses sequentially if the time series is independent.

Alternately, if there is a time series autocorrelated effect, you can rely on the general factorization of the likelihood

$$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2 | Y_1) \cdot \ldots \cdot Pr(Y_n | Y_{n-1} \ldots, Y_2, Y_1)$$

So by simply modeling the autocorrelation using some form of ARIMA or something, you can run a sequential analysis.

• This is a generic description of Bayesian updating. I am under the impression that the OP is looking for a specific reply to her/his [possibly unclear] question. Jan 15, 2016 at 8:53
• yes, i didnt quite follow the above reply to my question. To rephrase what i'm looking for - My confusion is do I need to update my previous days' data both in the likelihood and in the prior? I'm using informative priors as previous days' data from the AB test result. But i'm also using the likelihood as the cumulative conversion data. So my doubt is am I updating previous information in both my likelihood & prior? Jan 15, 2016 at 17:39
• @rettib it depends on the question you're trying to answer. Both would be interesting for different reasons. Using the informative prior gives you conditional inference on theta, E.g. Given the evidence suggests the odds ratio has this distribution, the present data give us belief about the distribution of theta. Jan 15, 2016 at 18:44