Multi test correction and hypothesis testing

Question

I have 9 datasets with one predictor and one target attribute. For each of the dataset, I am testing for a single hypothesis - whether the attributes are associated. I have got the following based on the test-statistic:

• Uncorrected p-values: 8 out of 9 p-values are significant ($p\le\alpha$)
• Bonferroni correction (FWER): 3 out of 9 p-values are significant ($p\le\alpha_{corrected}$)
• Benjamini–Hochberg correction (FDR): 6 out of 9 p-values are significant ($p\le\alpha_{B\&H}$)

I could combine 9 datasets but I am testing for each dataset separately because the context of the data in each dataset is important.

Question: Based on these findings, should I accept or reject the null hypothesis (the 2 attributes are not correlated?) and what could be the formal reasoning behind that?

The model is expected to produce few FP/FN but we are not sure to which extent. So we can allow a few errors from the model.

Answer 1

If you are interested in an association between the same predictor & target in all the datasets, you are using your dataset inefficiently by doing independent tests in each of them. Instead, consider using a mixed-model (a.k.a. hierarchical model) with dataset as a random effect (random intercept or random intercept + slope).

Essentially, it is a regression that models a single overall effect (for which you can compute a p-value) while allowing that effect to differ between groups (the datasets, in your cases). The differing contexts of the individual datasets is accounted for this way. This approach benefits from partial pooling of the datasets, as opposed to complete pooling, in which you merge all the individual datasets into one, undifferentiated large dataset. As a result, you use your data more efficiently and this has the side benefit of avoiding any confusion around corrections for multiple comparisons.