What is multiple testing and when to use Bonferroni Correction?

Question

I am confused about the concept of multivariate tests.

Let's say we have two use cases:

The 1st use case:

Run a test with different homepage background color

• group 1(control): green color
• group 2(treatment): red color
• group 3(treatment): blue color
• group 4(treatment): yellow color

The 2nd use case:

Run a test with different homepage backgroud color and different click button size

• group 1(control): green color and the small size button
• group 2(treatment): green color and big size button
• group 3(treatment): red color and small size button
• group 4(treatment): red color and big size button

The 3rd use case(similar to the 1st use case): Run 3 tests with different homepage background color

test 1:

• group 1(control): green color
• group 2(treatment): red color

test 2:

• group 1(control): green color
• group 2(treatment): blue color

test 3:

• group 1(control): green color
• group 2(treatment): yellow color

My question: which use case is the multivariate testing? If significant confidence is 95%, then what $p$-value should be used to evaluate these two tests?

My follow up question: I am wondering what is the Ho hypothesis and Ha hypothesis in these three use cases?

Usually, in an A/B test (1 control and 1 treatment), the null hypothesis is there is no significant difference between green color and red color. The alternative hypothesis is there is a significant difference between green color and red color.

What's the hypothesis in the multivariate testing?

Answer 1

Multiple testing and multivariate testing are different. Multiple testing is testing a hypothesis several times. Multivariate test is changing more than one variable between variants. Your second use case is an example of multivariate testing, where both homepage background and button size is changed.

Bonferroni correction is used when you repeat a test several times (either first one or the second). Let's say you want to test some change and decide based on its $p$-value. If you repeat the test $M$ times, likelihood of getting a $p$-significant event increases. Bonferroni correction adjusts the p-value threshold as $p/M$ so that this likelihood stays at reasonable levels.

Answer 2

Both scenarios have multiple testing.

Scenario 1, you have three different test groups being compared to the control group. For this scenario, I would use the Bonferroni adjustment by dividing the overall significance level by 3.

In Scenario 2, it seems like there are only two different comparisons you are interested in. First, is there a difference between red color and green color. Second, is there a difference between big button and small button. Therefore, I would use Bonferroni correction by dividing the significance level by 2 in order to test these two hypotheses. You could also be interested in the comparison of some of the combinations and then you would have to adjust it differently.

Bonferroni adjustment is not the only way to control the familywise error rate. In scenario 1 for example, Dunnett's test could be used. That would be more powerful and still control the familywise error rate.

Follow-up:

The global null hypothesis is when all simple null hypotheses are true. In your first scenario:

no significant difference between green color and red color
-and-
no significant difference between green color and blue color
-and-
no significant difference between green color and yellow color

The familywise error rate is the maximum probability of rejecting at least one true null hypothesis considering all possibilities of which are true and which are not. For example, when the global null hypothesis is true.

But, for the familywise error rate, you also consider the probability of rejecting the first one {no significant difference between green color and red color} when that is the only true null hypothesis; either of the two when exectly two of them are true; etc.

The Bonferroni correction controls the familywise error rate. In other words, the error rate is less than or equal to $\alpha$ regardless of which null hypotheses are true and regardless of how the p-values depend on each other.