Imagine we are performing 10 comparisons on a single variable of choice. The 10 respective experiments have different statistical powers. Now imagine that the least powerful and the most powerful tests both give a p-value of 0.05. Understandably when adjusting for multiple comparisons using let's say fdr method their q-values would be equal. While we already know that the significance from the weaker experiment must have had a stronger effect compared with the strong test which would trigger by the drop of the hat. I think, before multiple comparisons, the p-values must be adjusted for statistical power of the experiments so that the weaker experiment adjusted p-value would be less than the stronger experiment. Does anyone know the methodology for this?
closed as unclear what you're asking by gung♦ Dec 12 '18 at 16:04
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
I think that if you are testing the same phenomenon using different measures, it is not multiple comparisons; it is replication. You are not testing different things and trying to compensate for the possibility that some of the inferences could be wrong by accident. Your multiple tests reinforce one another.
Perhaps the question can best be addressed using meta-analysis tools (which I know little about).