# Correct usage/understanding of Bayes Factor when comparing two proportions

I'm just starting to learn R and explore Bayesian statistics, but I keep getting tripped on using Bayes Factor and (honestly), I'd love a little confirmation if my process is correct in interpreting difference between two sample proportions. I'd definitely appreciate any feedback.

Below is a scenario I'm looking at, and I'd love to know if the process below in R is right or if I'm applying anything incorrectly.

Scenario: Web users are split into two groups to test how well a website design performs. Since the designs are new, I'll use an uninformed prior (1,1). After running the split test for a few days, I have the following results. Results can only be successes/failures, so looking to use a binomial-beta.

Data:

Sample 1:

• n = 1000
• successes = 20
• failures = 980

Sample 2:

• n = 2000
• successes = 30
• failures = 1970

I want to use the following code to compare the resulting distributions and then determine the Bayes Factor.

theta1=rbeta(10000,20+1,980+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,30+1,1970+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf = theta_dif/(1-theta_dif) #This is the step I'm most unsure of is taking the probability that sample 1 is better over the complement (which is the probability that No. 2 is better) the right way to get a Bayes Factor?


Am I correct in these steps? In this case, the result comes out to be 5.01 which would be "Substantial". The 5.01 represents the roughly 85% / 15% from theta_dif (posterior probability that sample1 is better) divided by its compliment (theta2). If I do (85/15)/(15/85) the number is about 33, but is that correct Bayes Factor?

• Yeah. Uniform(1,1) is what I’m putting down for the prior in this example. In terms of the literature, I’m honestly a bit confused and hoping folks here can help me figure out how to judge the outcome of the sample testing. How do I compare the 85/15? – JAB Feb 4 at 3:37

The Bayes factor for testing $$H_0:\ p\le q$$ when $$X\sim\mathcal{N}(n,p)$$ and $$Y\sim\mathcal{N}(m,p)$$ is$$\mathfrak{B}_{01}(x,y)=\dfrac{\int_{p\le q} f_X(x|p)f_Y(y|q)\pi(p,q)\text{d}p\text{d}q}{\int_{p\ge q} f_X(x|p)f_Y(y|q)\pi(p,q)\text{d}p\text{d}q}=\dfrac{\int_{p\le q} p^x(1-p)^{n-x}q^y(1-q)^{m-y}\text{d}p\text{d}q}{\int_{p\ge q} p^x(1-p)^{n-x}q^y(1-q)^{m-y}\text{d}p\text{d}q}$$ which happens to be $$\mathfrak{B}_{01}(x,y)=\dfrac{\mathbb{P}(p\le q|x,y)}{\mathbb{P}(p\ge q|x,y)}$$ indeed. Hence, the R code in the question is correct.