I'm just starting to learn R and explore Bayesian statistics, but I keep getting tripped on using Bayes Factor and (honestly), I'd love a little confirmation if my process is correct in interpreting difference between two sample proportions. I'd definitely appreciate any feedback.
Below is a scenario I'm looking at, and I'd love to know if the process below in R is right or if I'm applying anything incorrectly.
Scenario: Web users are split into two groups to test how well a website design performs. Since the designs are new, I'll use an uninformed prior (1,1). After running the split test for a few days, I have the following results. Results can only be successes/failures, so looking to use a binomial-beta.
- n = 1000
- successes = 20
- failures = 980
- n = 2000
- successes = 30
- failures = 1970
I want to use the following code to compare the resulting distributions and then determine the Bayes Factor.
theta1=rbeta(10000,20+1,980+1) #taking 10,000 random draws from distribution with sample 1 theta2=rbeta(10000,30+1,1970+1) #taking 10,000 random draws from distribution with sample 2 theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2 bf = theta_dif/(1-theta_dif) #This is the step I'm most unsure of is taking the probability that sample 1 is better over the complement (which is the probability that No. 2 is better) the right way to get a Bayes Factor?
Am I correct in these steps? In this case, the result comes out to be 5.01 which would be "Substantial". The 5.01 represents the roughly 85% / 15% from theta_dif (posterior probability that sample1 is better) divided by its compliment (theta2). If I do (85/15)/(15/85) the number is about 33, but is that correct Bayes Factor?