1
$\begingroup$

I'm fairly new to bayesian. I'm trying to edit a bayesian python code for $A/B$ test analysis. I'm using uninformative priors as a beta distribution, so my $\alpha$ & $\beta$ parameters are $1$ & $1$ for both control & test for the first observation of the data. I have a function which takes in priors, visitors for control, visitors for test and their respective conversions to calculate the posterior

I'm using this Bayesian formula to update the priors -> $\textrm{Beta}(\alpha, \beta)$, and add the successes from the data, $x$, to $\alpha$ and the failures, $n – x$, to $\beta$, and there’s your posterior, $\textrm{Beta}(\alpha+x, \beta+n-x)$. Since this is for an $A/B$ test, i'm using cumulative visitors & conversions for each day as my likelihood and updating the priors from the formula above.

My question is, Should I use cumulative visitors & conversions for each day as my likelihood or visitors & conversions for each day separately since I'm updating the previous days' data in priors?

So my doubt is am I updating previous information in both my likelihood & prior?

$\endgroup$
  • $\begingroup$ What is the probability to see two independent questions on that topic on X validated within 10 hours?! $\endgroup$ – Xi'an Jan 15 '16 at 8:52
0
$\begingroup$

If the goal is to obtain $Pr(\theta | Y_1, Y_2, \ldots, Y_n)$ where the $Y$s are the cumulative data, then a simply application of Bayes Rule shows that:

$$Pr(\theta | Y_1, Y_2, \ldots, Y_n) = \frac{Pr( Y_1, Y_2, \ldots, Y_n | \theta) Pr(\theta)}{Pr(Y_1, Y_2, \ldots, Y_n)}$$

And under the assumption of independence of the $Y$s (being consecutive observations across time):

$$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2) \cdot \ldots \cdot Pr(Y_n)$$

and also

$$Pr(Y_1, Y_2, \ldots, Y_n | \theta) = Pr(Y_1| \theta) \cdot Pr( Y_2| \theta) \cdot \ldots \cdot Pr(Y_n| \theta)$$

So you see that yes you can run the analyses sequentially if the time series is independent.

Alternately, if there is a time series autocorrelated effect, you can rely on the general factorization of the likelihood

$$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2 | Y_1) \cdot \ldots \cdot Pr(Y_n | Y_{n-1} \ldots, Y_2, Y_1)$$

So by simply modeling the autocorrelation using some form of ARIMA or something, you can run a sequential analysis.

$\endgroup$
  • $\begingroup$ This is a generic description of Bayesian updating. I am under the impression that the OP is looking for a specific reply to her/his [possibly unclear] question. $\endgroup$ – Xi'an Jan 15 '16 at 8:53
  • $\begingroup$ yes, i didnt quite follow the above reply to my question. To rephrase what i'm looking for - My confusion is do I need to update my previous days' data both in the likelihood and in the prior? I'm using informative priors as previous days' data from the AB test result. But i'm also using the likelihood as the cumulative conversion data. So my doubt is am I updating previous information in both my likelihood & prior? $\endgroup$ – rettib Jan 15 '16 at 17:39
  • $\begingroup$ @rettib it depends on the question you're trying to answer. Both would be interesting for different reasons. Using the informative prior gives you conditional inference on theta, E.g. Given the evidence suggests the odds ratio has this distribution, the present data give us belief about the distribution of theta. $\endgroup$ – AdamO Jan 15 '16 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.