Clarification on Bayesian hypothesis testing

Question

I understand the mathematics behind Bayesian hypothesis testing, however I'm struggling to find many examples, specifically ones that apply to the problem I'm working on and was hoping to get some help here.

I'm trying to compare two solutions to a problem to help some design work. One of my solutions already exists and will be used a baseline S_b. I don't have any prior data as S_b hasn't been used to solve this particular problem before, but it has been used in similar ways. I have a new solution S_n which I want to compare to S_b to find if it performs significantly better.

My model is a simple success/fail simulation. I run S_b through the simulation a few hundred times with random seeds and get a binomial distribution of results. I then use an uninformed Beta prior, and apply Bayesian inference to get my Beta posterior, with a maximum likelihood at L_b. I then repeat this with my new design S_n, and get a likelihood of L_n.

I'm able to calculate my credible intervals, and can use the double integral to find the probability that S_n has a higher mean success rate than S_b. My question is, how do I test the hypothesis that S_n has a higher probability of success than S_b?

I'm aware of the Bayes factor, but I'm not sure what my two models should be. My current ideas are:

1. Use the probability that S_n has a higher mean success rate than S_b, and set a significance threshold. If P(L_n > L_b) > 0.95 then I can say that S_n has a greater chance of success than S_b.
2. Use the maximum likelihood of S_b as my null hypothesis and the max likelihood of S_n as my alternative. This feels like the cart driving the horse though so I'm sceptical of this approach.
3. Find at what level the credible intervals overlap (e.g. the credible intervals overlap for any value over 87%, so I am 87% certain that S_n will perform better than S_n).

Are any of these valid approaches, or am I way off target? I'm not a statistician by training and I've been buried in stats for the past few days so I may be overthinking my approach. Thank you in advance for any help you can give.

Answer 1

If you want to test a hypothesis (e.g., "the proportion of success is higher for $S_n$ than for $S_b$" vs. "the proportion of success is the same or lower for $S_n$ than for $S_b$"), Bayes Factors are one way to go. Have a look at Bayes Factors for contingency tables, this sounds as if this could match your scenario: https://richarddmorey.github.io/BayesFactor/#ctables