# Multi test correction and hypothesis testing

I have 9 datasets with one predictor and one target attribute. For each of the dataset, I am testing for a single hypothesis - whether the attributes are associated. I have got the following based on the test-statistic:

• Uncorrected p-values: 8 out of 9 p-values are significant ($$p\le\alpha$$)
• Bonferroni correction (FWER): 3 out of 9 p-values are significant ($$p\le\alpha_{corrected}$$)
• Benjamini–Hochberg correction (FDR): 6 out of 9 p-values are significant ($$p\le\alpha_{B\&H}$$)

I could combine 9 datasets but I am testing for each dataset separately because the context of the data in each dataset is important.

Question: Based on these findings, should I accept or reject the null hypothesis (the 2 attributes are not correlated?) and what could be the formal reasoning behind that?

The model is expected to produce few FP/FN but we are not sure to which extent. So we can allow a few errors from the model.

• Well, the two attributes are significantly different in 6 out of 9 data sets, is what I would say. – user2974951 Sep 4 '19 at 6:27

If you are interested in an association between the same predictor & target in all the datasets, you are using your dataset inefficiently by doing independent tests in each of them. Instead, consider using a (a.k.a. hierarchical model) with dataset as a random effect (random intercept or random intercept + slope).