Please consider the following experiment:
Research question:
Is there an association between Water Quality and Number of Creatures living in lakes in the US.Water Quality is defined by 10 indices
(e.g., ph, lead count, etc.).For creatures the number of 10 different common creatures was counted
(e.g., frogs, bass, etc).Water samples and creature counts were taken from 25 lakes.
For statistical analysis we used linear regression with number of creatures as outcome measures (10 species) and water quality as predictor (10 indices), totaling to 100 tests.
At an alpha level of 0.05 I would expect 5 of these tests to be significant on the basis of chance.
To correct for that, I could use a Bonferroni corrected alpha level of alpha/n=0.0005.
Now let's assume that I observed 20 significant associations at an uncorrected alpha level of p<0.05, but all the individual p-values are 0.0005<p<0.05. Thus, no significant associations remain after Bonferroni correction for multiple comparisons.
Question:
Is the fact that the number of times H0 was rejected meaningful considering that this number was larger than expected (i.e., 20 observed vs. 5 expected)?
(With meaningful, I mean: does it support my overal hypothesis that water quality and number of creatures living in it are associated).
Note: I am not looking for alternative potentially less stringent correction options such as Bonferroni-Holmes correction, FDR correction, q-values, etc. I am also not looking for ways to combine variables etc. to reanalyze the data.