# Multiple testing adjustments for Bayes Factors

I manage an platform that has succesfully gotten a Bayesian approach to experimentation in production.

One new feature we want to implement is to do inference for multiple metrics (e.g. conversion rate and revenue). I know that one way to solve this is a model to infer paramenters for all metricss e.g. follow the approach of model from this. However, this is impractical for our use as we have ~100s of , and want to do inference on 5-6.

Below is an small example of what I am thinking about. (Adapted from this question.)

Scenario: Web users are split into two groups to test if users cancel less under the new design. Since the designs are new, I'll use an uninformed prior (1,1). After running the split test for a few days, I have the following results. Results can only be cancel/not-cancel ('success'/'failure'), so looking to use a binomial-beta.

Data:

Sample 1:

n = 1000
cancel = 20
don't-cancel = 980


Sample 2:

n = 2000
cancel = 30
don't-cancel = 1970


But in this case we also want to measure pauses on the same set of users. In that case we have the following set of data:

Sample 1:

n = 1000
pauses = 50
non-pauses = 950


Sample 2:

n = 2000
pauses = 130
non-pauses = 1870


I can use the following code to calculate the bayes factors:

Cancelation data:

theta1=rbeta(10000,20+1,980+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,30+1,1970+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf_1 = theta_dif/(1-theta_dif)


Pause data:

theta1=rbeta(10000,50+1,950+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,130+1,1870+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf_2 = theta_dif/(1-theta_dif)


My question is: is there any post-inference proceedure I need to do to adjust my Bayes factors (or theta_diff), to account for reusing the data and doing multiple tests on it?

E.g. I could divide theta_diff by 2, meaning the Bayes factor calculation now is: (with $$p = \text{theta_diff}$$)

$$\frac{p/2}{1 - p/2} = \frac{p}{2 - p}$$

I'm honestly confused

There's not trivial way (like the Bonferroni adjustment for p-values - NB: of course there's uniformly more powerful frequentist multiple testing adjustments) for adjusting Bayes factors in a way that would control the familywise type I error rate. You can see this, because as discussed in this paper - if we ignore prior distributions - the Bayes factor calculated from a p-value for a hypothesis test comparing two parameter values is given by $$BF(p)=1/f(p|H_A)$$, where $$f(p|H_A)$$ is the average likelihood $$f(y|\theta_A)$$ averaged over the prior distribution for $$\theta$$ under the alternative $$H_A$$. This likelihood depends also on the power/size of your experiment.
I guess for a given prior and design, you can calculate $$f(p|H_A)$$. If not analytically, then at least by simulation from parameter values under the alternative followed by simulating the experiment and its analysis, and treating the posterior probability that the value of interest is (greater/smaller) than the null hypothesis value. You can then approximate the likelihood for this probability (or its logit) via e.g. a kernel density estimate and see where the value obtain for your actual data lies relative to it. But, that does seem rather roundabout...