When to Apply P-Value Corrections for Multiple Testing: A Multi-Outcome Study Example

Question

Suppose I want to examine the association between a total score and independent binary variables (e.g., sex) and multi-categorical variables (e.g., grade). I conducted a t-test for the binary variables and ANOVA for the multi-categorical variables. Additionally, I created a correlation matrix between the total score and the sub-scores. In the end, I had six tables with each score as the dependent variable and the variables of interest (e.g., sex, grade) as the independent variables, in addition to the correlation matrix.

For the significant ANOVAs, I planned to conduct post hoc tests using Tukey’s test and Dunnett’s test, depending on the variable. However, I encountered an issue: these post hoc tests adjust p-values based on the number of comparisons. If I adjust for p-values, shouldn’t I also adjust for all the tests I have conducted in the study, including correlations and t-tests? In this context, post hoc tests may not be entirely relevant, as I should adjust for all tests, not just the post hoc comparisons. Wouldn’t it be more reasonable to conduct pairwise t-tests and use a correction method like Bonferroni to account for all the tests performed, not just the post hoc tests?

It seems that post hoc tests are only relevant if I had a single hypothesis tested using ANOVA, as they adjust for the overall p-value in this case. However, most studies in the literature use post hoc tests after ANOVA without adjusting for other tests they have conducted.

I want to know if my understanding is correct or if there is a better approach, as I think using Bonferroni correction is more conservative than other post hoc tests.

Edit: After reviewing this paper https://doi.org/10.1016/j.metip.2023.100120 the author argues that correction should only be used in the context of disjunction testing

Corrections for multiple testing serve the purpose of making in­ferences about an omnibus null hypothesis with a pre-specified Type-I error rate α when the omnibus null can only be tested piecewise through a set of k surrogate nulls.

A Type-I error rate is associated with each statistical claim of sig­nificance and multiple statistical claims of significance in a research report have each its own Type-I error rate. These Type-I error rates are independent from one another and co-exist. A correction for multiple testing is necessary only when testing a joint intersection null hypothesis, that is, when the sole claim about it can only be made after appli­cation of several statistical tests, each of which must be conducted at a suitably determined α* to ensure that the Type-I error rate for the joint intersection null is the declared α. Outside this specific context, cor­rections for multiple testing only come down to testing statistical hy­pothesis at some α* < α and, thus, to departing from the declared alpha level despite the fact that such corrections are usually purported to keep the Type-I error rate at α.

Also, it seems that the conditional corrections for multiple testing only after significant ANOVAs may not be appropriate. I was thinking that it may be accurate in situations using Dunnett’s test as we are focusing already in the post-hoc results, but in situations using Tukey’s test it may be the main outcome is the ANOVA results and that post-hoc here serves as an additional outcome ( we do not have specific hypothesis to look for but rather we will be interested in the main ANOVA without focusing too much on the post-hoc results). Apart from that, there is still the problem of inflation of type-I error; however, I need to search more about it in the context of Tukey’s test and Dunnett’s test as the stimulation in this study was done in the context of Bonferroni correction.

In our review of empirical practices, conducting pairwise compari­sons to follow up on a significant ANOVA was the most common sce­nario in which corrections for multiple testing were used. Yet, such corrections should not be used here and not just because of the implied conditional approach that greatly inflates Type-I error rates (see red 12 symbols in Fig. 1). To begin with, an ANOVA is already a one-shot test of the omnibus joint intersection null and there is no need to test it again via surrogates with a correction for multiple testing.12 More impor­tantly, pairwise comparisons imply turning attention to individual nulls considered separately from the others and for which the declared Type-I error rate must also hold individually. In the vast majority of cases, the research question does not even call for a joint intersection null, that is, there is hardly ever a proposition that all means should be equal such that if the proposition is rejected in its entirety no further question arises. Far from this, there is usually a surmise that one or more particular means will be different from the rest, which places the focus directly on the pairwise comparisons with no omnibus null implied. If this is the case, direct pairwise comparisons (without a preliminary ANOVA) is the only advisable approach, with each comparison tested at α without a correction for non-existent multiple testing of the same proposition.

Answer 1

The question is framed as if Bayesian statistics does not exist. It is very useful to look at Bayesian solutions to the problem, to better understand assumptions, context, and the meaning of multiplicities. For the problem at hand, I would consider doing two analyses. The first would use a Bayesian ordinal regression model to assess evidence for differences in the overall score. The second would be a multivariate model where all the subscores are modeled as similar semiparametric ordinal models, with dependencies modeled through random effects or a copula. Then one can obtain, for example, posterior probabilities that

• any subscore is affected by more than a trivial amount
• all subscores are affected by any amount
• at least 2 subscores are affected by any (or a non-trivial) amount

None of these evidentiary assessments involve multiplicities; they just involve concisely stating compound assertions of interest.

See this for more information.

Answer 2

You say your goal is to assess "the association between a total score and its sub-scores". By definition, the subscore is associated with the total. How can quantifying this association be helpful?