Is it necessary to do a Bonferroni correction on a) exploratory analysis b) correlations or c) if there are 3 different dependent variables

Question

My study is a 2x2 within design and I have a series of 8 hypotheses. These include one ANOVA , one t-test and 4 Pearson correlations. However, the ANOVA and 3 of the correlations include the same dependent variable (s ratings), and then the t-test and the other correlations include a different dependent variable (d ratings).

Additionally, I have performed some additional exploratory analysis including an ANOVA and 3 t-tests. The ANOVA has (s ratings) as a dependent variable and the t-tests have (d ratings) as a dependent variable.

So overall there are 5 hypotheses/results for a DV of s ratings, and 6 hypotheses/results for a DV for r ratings.

So my questions are:

1. Do exploratory and confirmatory analysis get combined to create the Bonferroni correction?

2. Do I do two separate Bonferroni corrections as they relate to different DVs?

3. Do correlations need a correction as they technically don't have a DV?

Answer 1

Bonferroni correction is used to address the problem of multiple testing which becomes an issue when more than one hypothesis test ought to be conducted. Maybe it helps for you to think about: why is multiple testing a problem at all?

If you test a hypothesis, often you are going to accept any p-values below 0.05 as "significant", meaning that you reject the null hypothesis. We are following this rule to be able to say:

If actually the null hypothesis is true, there is not more than 5% chance that I reject it.

Or, to put it the other way:

If actually the null hypothesis is true, there is 95% chance that I correctly do not reject it.

If there are two hypotheses that you are testing and you use the same rule, this statement is no longer true. Instead, for the first hypothesis there is 95% chance to correctly not reject it, and for the second hypothesis there is 95% chance to correctly not reject it. So the probability of doing no false rejection is 95% * 95% = 90.25%, and the probability of doing at least one false rejection is 100% - 90.25% = 9.75%.

Bonferroni correction ensures that with several hypothesis tests you have still this 5% probability of wrongly rejecting at least one null hypothesis. But you pay a price: you lose power, so the probability of correctly rejecting the null hypothesis diminishes.

Now, it depends on what you are aiming for: do you want to be able to say that there is not more than 5% chance that you have wrongly rejected any of the null hypotheses? Or are you in an explorative setting, where your actual hypotheses do not matter that much and you are rather interested in "what you can find in the data"?

To address your questions directly:

1. no, exploratory analyses are not used for testing a hypothesis, hence p-values are not interpreted in terms of "significance", hence Bonferroni correction does not apply
2. It is not the DV that matters but the hypothesis that you are testing. Are the hypotheses completely separated? Do you care about being able to state "there is not more than 5% chance to wrongly reject any of the two hypotheses"?
3. Yes, what matters is the number of hypothesis tests (i.e. the number of p-values you are checking whether they are smaller than 0.05).