My question anonymizes the 'nouns' of the question to protect my employer. It is not really about lab rats and experimental treatments.
I am also coming more from a Machine Learning background, so my lingo might reflect that, although I make a honest effort to do the Statistics correctly and use the right terminology.
1,000 lab rats each receive one of 26 experimental treatments labeled A, B, ..., Z. The treatments are very unequally distributed - some drugs were administered to only a few mice, and others to hundreds of mice.
Rats whose blood tests showed significant improvement after two weeks were marked as "Positive Outcome", otherwise they are marked as "Negative Outcome".
To determine which treatments have some kind of association with the outcome, I have constructed 26 separate 2x2 contingency tables which compare "This Treatment" (i.e.; Treatment A) and "Other Treatments" (i.e.; Treatment B-Z) vs Outcome. I do tests for association at the 0.05 significance level.
But wait! Aren't we supposed to use the Bonferroni Correction for multiple testing using the 0.05 / 26 = 0.0019 significance level? Sure, but then nothing is statistically significant, and I know based on domain expertise that this is not a practically useful or "accurate" conclusion.
But because of the lack of independence of the tests, I think that a less conservative correction would still guarantee a false positive rate among all tests.
The tests are not independent - a successful, very frequently administered treatment will be in the "rest" of 25 of the "one-vs-rest" hypotheses.
Looking into other approaches, I don't want to do something exotic like q-value testing (which controls for False Discoveries instead of False Positives) because:
- It limits my ability to communicate the results because it is a less common approach
- There is a far greater cost to the organization of a False Positive - that seems to be the thing to avoid.
So, I'd like to use a p-value, but I'd just like a correction that reflects reality a bit better than the Bonferroni correction. One that takes into account the lack of independence between the multiple comparisons, for example. Or just avoids the problem altogether.
Do you have recommendations? Permutation Testing seems like it might be a good choice.