2
$\begingroup$

Please consider the following experiment:

  • Research question:
    Is there an association between Water Quality and Number of Creatures living in lakes in the US.

  • Water Quality is defined by 10 indices
    (e.g., ph, lead count, etc.).

  • For creatures the number of 10 different common creatures was counted
    (e.g., frogs, bass, etc).

  • Water samples and creature counts were taken from 25 lakes.

  • For statistical analysis we used linear regression with number of creatures as outcome measures (10 species) and water quality as predictor (10 indices), totaling to 100 tests.

At an alpha level of 0.05 I would expect 5 of these tests to be significant on the basis of chance.

To correct for that, I could use a Bonferroni corrected alpha level of alpha/n=0.0005.

Now let's assume that I observed 20 significant associations at an uncorrected alpha level of p<0.05, but all the individual p-values are 0.0005<p<0.05. Thus, no significant associations remain after Bonferroni correction for multiple comparisons.

Question:
Is the fact that the number of times H0 was rejected meaningful considering that this number was larger than expected (i.e., 20 observed vs. 5 expected)?


(With meaningful, I mean: does it support my overal hypothesis that water quality and number of creatures living in it are associated).



Note: I am not looking for alternative potentially less stringent correction options such as Bonferroni-Holmes correction, FDR correction, q-values, etc. I am also not looking for ways to combine variables etc. to reanalyze the data.

Improve this question
$\endgroup$
5
  • $\begingroup$ It's unclear what "these tests" refers to--your describe data but don't appear to specify any tests at all. It's not even clear how many tests you performed, nor how the data used in each one might be interrelated. Could you supply that information? BTW, what are "create counts"? $\endgroup$
    – whuber ♦
    Aug 4, 2016 at 22:08
  • 1
    $\begingroup$ Thanks for your comments. I have added the information on the statistical tests used and corrected the typo (create counts > creature counts). $\endgroup$
    – Vincent
    Aug 4, 2016 at 22:30
  • $\begingroup$ That helps (I didn't expect "create" to become "creature"!). Have you assessed the correlations among the ten response variables (the creature counts)? How about the correlations among the ten regressor variables (water quality)? Evaluation of these correlations ought to play a large role in assessing just how many independent results you actually have. $\endgroup$
    – whuber ♦
    Aug 4, 2016 at 22:35
  • $\begingroup$ Thanks for helping. I actually have looked at the correlations! Water quality measures are correlated as well as the creature counts. So I do know that these associations are not totally independent. $\endgroup$
    – Vincent
    Aug 4, 2016 at 22:45
  • $\begingroup$ I know you're not looking for alternative approaches, but the nature of your data suggest you consider a multivariate model in which you fit all the counts simultaneously. $\endgroup$
    – whuber ♦
    Aug 4, 2016 at 23:13

2 Answers 2

Reset to default
2
$\begingroup$

  1. There's no such thing as "proof of H1" in null hypothesis testing. Low p-values may be supporting evidence, but not proof.

  2. You say in your question that you are not interested in using alternatives to standard Bonferroni correction or in using an alternative testing structure. In that case, with no p-values < .0005, I suppose you've answered your own question: nothing is statistically significant, no matter how many p-values are below .05, because .05 isn't the Bonferroni-adjusted threshold! As you noted, the relevant threshold is .0005.

  3. More generally, it might be worth noting that 20/100 p-values were < .05, but without knowing how correlated those tests are, it's hard to say how remarkable those simultaneous significances are. For instance, imagine the 20 tests are very highly positively correlated with each other. Then any time one test is significant (either due to a real effect or due to Type I error) the other tests will likely be significant too, so the fact that they were simultaneously significant in any particular case would be nothing to write home about. On the other extreme, if the tests are perfectly independent, then the probability of all 20 tests being < .05 purely by "chance" at the same time is quite low. Clearly, the simultaneous significance of 20 tests is more remarkable in the latter case.

Improve this answer
$\endgroup$
1
  • $\begingroup$ nice match between user name and answer ... $\endgroup$
    – Ben Bolker
    Oct 23, 2016 at 3:04
0
$\begingroup$

In this case, since you're running regression models on individual water quality measures against individual species, the p-values of the individual tests may not signify an overall relationship.

For example, let's say that you find that the model with lead count against the number of bass turns out to show a statistically significant relationship, with the number of bass positively effecting the lead count in the lake. Let's say you also find out that with another species (maybe frogs) has a statistically significant relationship against lead count, but in the other direction. That is, there is a negative slope, and the more frogs there are the less the lead count is (this might be totally incorrect based on biology, but its just an example.)

Both of these tests end up stating that their associates species has a statistically significant relationship with the lead count, but in opposite directions. This could potentially cause the effects to cancel out. As such, it wouldn't really make sense to say that overall Water Quality has an association with the Number of Creatures living in lakes in the U.S, even if 20 of the individual 1-on-1 regression models returned significant results.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In what sense do you mean that "effects cancel out"? One would think that a finding like your hypothetical--that water quality is associated with greater numbers of one species and fewer numbers of another species--would be important and interesting. If the increase and decrease "cancel," then why count species separately? $\endgroup$
    – whuber ♦
    Aug 5, 2016 at 14:13
  • $\begingroup$ I meant something like, with the models above, you're finding effects between individual species against individual water quality measures, and you can't say that because you found several such individual relationships, that there is a relationship between overall water quality and overall species count $\endgroup$
    – Green
    Aug 5, 2016 at 16:56
  • $\begingroup$ Sure, but what if these associations all do point in the same direction? $\endgroup$
    – Vincent
    Aug 9, 2016 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.