I'm just starting to learn R and explore Bayesian statistics, but I keep getting tripped on using Bayes Factor and (honestly), I'd love a little confirmation if my process is correct in interpreting difference between two sample proportions. I'd definitely appreciate any feedback.

Below is a scenario I'm looking at, and I'd love to know if the process below in R is right or if I'm applying anything incorrectly.

Scenario: Web users are split into two groups to test how well a website design performs. Since the designs are new, I'll use an uninformed prior (1,1). After running the split test for a few days, I have the following results. Results can only be successes/failures, so looking to use a binomial-beta.


Sample 1:

  • n = 1000
  • successes = 20
  • failures = 980

Sample 2:

  • n = 2000
  • successes = 30
  • failures = 1970

I want to use the following code to compare the resulting distributions and then determine the Bayes Factor.

theta1=rbeta(10000,20+1,980+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,30+1,1970+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf = theta_dif/(1-theta_dif) #This is the step I'm most unsure of is taking the probability that sample 1 is better over the complement (which is the probability that No. 2 is better) the right way to get a Bayes Factor? 

Am I correct in these steps? In this case, the result comes out to be 5.01 which would be "Substantial". The 5.01 represents the roughly 85% / 15% from theta_dif (posterior probability that sample1 is better) divided by its compliment (theta2). If I do (85/15)/(15/85) the number is about 33, but is that correct Bayes Factor?

  • $\begingroup$ Yeah. Uniform(1,1) is what I’m putting down for the prior in this example. In terms of the literature, I’m honestly a bit confused and hoping folks here can help me figure out how to judge the outcome of the sample testing. How do I compare the 85/15? $\endgroup$ – JAB Feb 4 '19 at 3:37

The Bayes factor for testing $H_0:\ p\le q$ when $X\sim\mathcal{N}(n,p)$ and $Y\sim\mathcal{N}(m,p)$ is$$\mathfrak{B}_{01}(x,y)=\dfrac{\int_{p\le q} f_X(x|p)f_Y(y|q)\pi(p,q)\text{d}p\text{d}q}{\int_{p\ge q} f_X(x|p)f_Y(y|q)\pi(p,q)\text{d}p\text{d}q}=\dfrac{\int_{p\le q} p^x(1-p)^{n-x}q^y(1-q)^{m-y}\text{d}p\text{d}q}{\int_{p\ge q} p^x(1-p)^{n-x}q^y(1-q)^{m-y}\text{d}p\text{d}q}$$ which happens to be $$\mathfrak{B}_{01}(x,y)=\dfrac{\mathbb{P}(p\le q|x,y)}{\mathbb{P}(p\ge q|x,y)}$$ indeed. Hence, the R code in the question is correct.

Note that in the question "uninformed (1,1)" should be either "Uniform(0,1)" or "Beta(1,1)".

  • 1
    $\begingroup$ ’an Thanks for prompt (and answer). I checked the box $\endgroup$ – JAB Feb 7 '19 at 0:47

Indeed you are now computing the BF of theta1 > theta2 against its complement, theta1 ≤ theta2. Note that if it may be that theta1 and theta2 are 'exactly' equal (this depends on the context) you would need to perform an precise test instead of a one-sided test. The choice of the prior (i.e., where do you expect theta1 and theta2 to lie if they would be unequal) would then affect the results. For the BF for the precise test you could evaluate the posterior and prior at the nul value (also known as the Savage-Dickey density ratio), i.e.,

theta1_post = rbeta(100000,20+1,980+1)
theta2_post = rbeta(100000,30+1,1970+1)
theta_dif_post = theta1_post-theta2_post
dens_post <- density(theta_dif_post, from=0, to=0, n=1)$y
theta1_prior = rbeta(100000,1,1)
theta2_prior = rbeta(100000,1,1)
theta_dif_prior = theta1_prior-theta2_prior
dens_prior <- density(theta_dif_prior, from=0, to=0, n=1)$y
BFprecise <- dens_post / dens_prior

For your data this results in a lot of evidence for equality of theta's (a BF of around 48). Note (again) that the choice of the prior matters more than for one-sided testing. The above computation is done when using uniform priors, which implies that before observing the data you assume that any value for the theta's in the interval (0,1) is equally likely a priori. Further note that the Savage-Dickey density ratio should be used with some care but here I think it is alright.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.