I'm just starting to learn R and explore Bayesian statistics, but I keep getting tripped on using Bayes Factor and (honestly), I'd love a little confirmation if my process is correct in interpreting difference between two sample proportions. I'd definitely appreciate any feedback.

Below is a scenario I'm looking at, and I'd love to know if the process below in R is right or if I'm applying anything incorrectly.

Scenario: Web users are split into two groups to test how well a website design performs. Since the designs are new, I'll use an uninformed prior (1,1). After running the split test for a few days, I have the following results. Results can only be successes/failures, so looking to use a binomial-beta.


Sample 1:

  • n = 1000
  • successes = 20
  • failures = 980

Sample 2:

  • n = 2000
  • successes = 30
  • failures = 1970

I want to use the following code to compare the resulting distributions and then determine the Bayes Factor.

theta1=rbeta(10000,20+1,980+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,30+1,1970+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf = theta_dif/(1-theta_dif) #This is the step I'm most unsure of is taking the probability that sample 1 is better over the complement (which is the probability that No. 2 is better) the right way to get a Bayes Factor? 

Am I correct in these steps? In this case, the result comes out to be 5.01 which would be "Substantial". The 5.01 represents the roughly 85% / 15% from theta_dif (posterior probability that sample1 is better) divided by its compliment (theta2). If I do (85/15)/(15/85) the number is about 33, but is that correct Bayes Factor?

  • $\begingroup$ Yeah. Uniform(1,1) is what I’m putting down for the prior in this example. In terms of the literature, I’m honestly a bit confused and hoping folks here can help me figure out how to judge the outcome of the sample testing. How do I compare the 85/15? $\endgroup$ – JAB Feb 4 '19 at 3:37

The Bayes factor for testing $H_0:\ p\le q$ when $X\sim\mathcal{N}(n,p)$ and $Y\sim\mathcal{N}(m,p)$ is$$\mathfrak{B}_{01}(x,y)=\dfrac{\int_{p\le q} f_X(x|p)f_Y(y|q)\pi(p,q)\text{d}p\text{d}q}{\int_{p\ge q} f_X(x|p)f_Y(y|q)\pi(p,q)\text{d}p\text{d}q}=\dfrac{\int_{p\le q} p^x(1-p)^{n-x}q^y(1-q)^{m-y}\text{d}p\text{d}q}{\int_{p\ge q} p^x(1-p)^{n-x}q^y(1-q)^{m-y}\text{d}p\text{d}q}$$ which happens to be $$\mathfrak{B}_{01}(x,y)=\dfrac{\mathbb{P}(p\le q|x,y)}{\mathbb{P}(p\ge q|x,y)}$$ indeed. Hence, the R code in the question is correct.

Note that in the question "uninformed (1,1)" should be either "Uniform(0,1)" or "Beta(1,1)".

  • 1
    $\begingroup$ ’an Thanks for prompt (and answer). I checked the box $\endgroup$ – JAB Feb 7 '19 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.