I understand the mathematics behind Bayesian hypothesis testing, however I'm struggling to find many examples, specifically ones that apply to the problem I'm working on and was hoping to get some help here.
I'm trying to compare two solutions to a problem to help some design work. One of my solutions already exists and will be used a baseline Sb. I don't have any prior data as Sb hasn't been used to solve this particular problem before, but it has been used in similar ways. I have a new solution Sn which I want to compare to Sb to find if it performs significantly better.
My model is a simple success/fail simulation. I run Sb through the simulation a few hundred times with random seeds and get a binomial distribution of results. I then use an uninformed Beta prior, and apply Bayesian inference to get my Beta posterior, with a maximum likelihood at Lb. I then repeat this with my new design Sn, and get a likelihood of Ln.
I'm able to calculate my credible intervals, and can use the double integral to find the probability that Sn has a higher mean success rate than Sb. My question is, how do I test the hypothesis that Sn has a higher probability of success than Sb?
I'm aware of the Bayes factor, but I'm not sure what my two models should be. My current ideas are:
- Use the probability that Sn has a higher mean success rate than Sb, and set a significance threshold. If P(Ln > Lb) > 0.95 then I can say that Sn has a greater chance of success than Sb.
- Use the maximum likelihood of Sb as my null hypothesis and the max likelihood of Sn as my alternative. This feels like the cart driving the horse though so I'm sceptical of this approach.
- Find at what level the credible intervals overlap (e.g. the credible intervals overlap for any value over 87%, so I am 87% certain that Sn will perform better than Sn).
Are any of these valid approaches, or am I way off target? I'm not a statistician by training and I've been buried in stats for the past few days so I may be overthinking my approach. Thank you in advance for any help you can give.