Imagine we are performing 10 comparisons on a single variable of choice. The 10 respective experiments have different statistical powers. Now imagine that the least powerful and the most powerful tests both give a p-value of 0.05. Understandably when adjusting for multiple comparisons using let's say fdr method their q-values would be equal. While we already know that the significance from the weaker experiment must have had a stronger effect compared with the strong test which would trigger by the drop of the hat. I think, before multiple comparisons, the p-values must be adjusted for statistical power of the experiments so that the weaker experiment adjusted p-value would be less than the stronger experiment. Does anyone know the methodology for this?


closed as unclear what you're asking by gung Dec 12 '18 at 16:04

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Is it the same response variable and treatments? Then it isn’t multiple comparisons, but the results of the experiments differ, which might be a concern. If different responses or treatments, then what exactly is the issue? $\endgroup$ – rvl Dec 11 '18 at 0:25
  • 1
    $\begingroup$ No. It is the same output measured through different means. I think of it like this. Trying to verify whether an earthquake happened by looking at tree branches moving and seismograph simultaneously. The first one is over sensitive but has high FDR but becomes significant easily. the latter is not sensitive but has a very low FDR. Now if I multiple comparison them, I don't think adjusting their p-values the same amount ( imagining both are let's say 0.05) is right. Seismograph significance is related to a much stronger effect. $\endgroup$ – Theoden Dec 11 '18 at 14:30
  • $\begingroup$ Are you trying to classify something (detect earthquakes) or test a scientific hypothesis? Are you trying to identify if each of a set of manifest variables is a function of some latent variable? Are you trying to estimate the best linear combination of your variables? Please give concrete details of your actual situation. What are your variables? What are your data? How much data do you have? What are your ultimate goals? Without that information, this cannot be answered. $\endgroup$ – gung Dec 12 '18 at 16:04
  • $\begingroup$ It's about testing multiple scientific hypotheses. Think about pathway analysis in RNA expressions for instance. You have one known controlled variable such as mutation status and then you have multiple pathways you are trying to test for enrichment. Clearly you need to control for multiple comparison here since there are numerous pathways. However if your significant detection method has different power on different pathways ( for any reason) the pathway with strong response becomes significant with a small effect but the one with weak response needs a huge effect. $\endgroup$ – Theoden Dec 12 '18 at 16:17
  • $\begingroup$ Considering above it is not sound to correct them the same amount if they both have the same p-value. The question is how to compensate for individual experiment power differences. $\endgroup$ – Theoden Dec 12 '18 at 16:17

I think that if you are testing the same phenomenon using different measures, it is not multiple comparisons; it is replication. You are not testing different things and trying to compensate for the possibility that some of the inferences could be wrong by accident. Your multiple tests reinforce one another.

Perhaps the question can best be addressed using meta-analysis tools (which I know little about).


Not the answer you're looking for? Browse other questions tagged or ask your own question.