3
$\begingroup$

I manage an platform that has succesfully gotten a Bayesian approach to experimentation in production.

One new feature we want to implement is to do inference for multiple metrics (e.g. conversion rate and revenue). I know that one way to solve this is a model to infer paramenters for all metricss e.g. follow the approach of model from this. However, this is impractical for our use as we have ~100s of , and want to do inference on 5-6.

Below is an small example of what I am thinking about. (Adapted from this question.)

Scenario: Web users are split into two groups to test if users cancel less under the new design. Since the designs are new, I'll use an uninformed prior (1,1). After running the split test for a few days, I have the following results. Results can only be cancel/not-cancel ('success'/'failure'), so looking to use a binomial-beta.

Data:

Sample 1:

n = 1000
cancel = 20
don't-cancel = 980

Sample 2:

n = 2000
cancel = 30
don't-cancel = 1970

But in this case we also want to measure pauses on the same set of users. In that case we have the following set of data:

Sample 1:

n = 1000
pauses = 50
non-pauses = 950 

Sample 2:

n = 2000
pauses = 130
non-pauses = 1870

I can use the following code to calculate the bayes factors:

Cancelation data:

theta1=rbeta(10000,20+1,980+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,30+1,1970+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf_1 = theta_dif/(1-theta_dif) 

Pause data:

theta1=rbeta(10000,50+1,950+1) #taking 10,000 random draws from distribution with sample 1
theta2=rbeta(10000,130+1,1870+1) #taking 10,000 random draws from distribution with sample 2
theta_dif = mean(theta1>theta2) #Find the posterior probability that someone from sample 1 will convert more than someone in sample 2
bf_2 = theta_dif/(1-theta_dif) 

My question is: is there any post-inference proceedure I need to do to adjust my Bayes factors (or theta_diff), to account for reusing the data and doing multiple tests on it?

E.g. I could divide theta_diff by 2, meaning the Bayes factor calculation now is: (with $ p = \text{theta_diff} $)

$$\frac{p/2}{1 - p/2} = \frac{p}{2 - p}$$

I'm honestly confused

$\endgroup$

1 Answer 1

1
+50
$\begingroup$

Whether you "need to" adjust depends on what you desire. Do you care about whether when you do many such "tests" repeatedly you would more frequently wrongly claim there is a difference when there isn't one because you looked at multiple metrics (=familywise type I error rate for your experiment)? If so, you should adjust (because you would otherwise get an inflated familywise type I error rate). If you instead say that on philosophical grounds you don't care about the type I error rate (some Bayesians would take that perspective appealing to the strong likelihood principle), then you don't. If you have some complicated composite decision criteria (e.g. at least 2 out of 3 metrics need to improve or the like, then it gets pretty complicated to understand the operating characteristics of such a rule, so I'll ignore that option).

There's not trivial way (like the Bonferroni adjustment for p-values - NB: of course there's uniformly more powerful frequentist multiple testing adjustments) for adjusting Bayes factors in a way that would control the familywise type I error rate. You can see this, because as discussed in this paper - if we ignore prior distributions - the Bayes factor calculated from a p-value for a hypothesis test comparing two parameter values is given by $BF(p)=1/f(p|H_A)$, where $f(p|H_A)$ is the average likelihood $f(y|\theta_A)$ averaged over the prior distribution for $\theta$ under the alternative $H_A$. This likelihood depends also on the power/size of your experiment.

I guess for a given prior and design, you can calculate $f(p|H_A)$. If not analytically, then at least by simulation from parameter values under the alternative followed by simulating the experiment and its analysis, and treating the posterior probability that the value of interest is (greater/smaller) than the null hypothesis value. You can then approximate the likelihood for this probability (or its logit) via e.g. a kernel density estimate and see where the value obtain for your actual data lies relative to it. But, that does seem rather roundabout...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.