7
$\begingroup$

I have done 350 independent tests. I need to correct them for the multiple comparisons. I am using the Benjamini-Hochberg (BH) procedure to control FDR. However choosing the Q (the level at which we want to control the FDR) beforehand could be restrictive (say e.g. I find no discovery for Q=0.05 and 40 for Q=0.1: it would be nice tell something about those 36 supposedly true discoveries made with Q=0.1 instead of reporting no discovery).

I have two questions:

  1. If I'm doing an exploratory analysis, is it a wrong procedure to have a look at the number of discoveries for varying levels of Q instead of fixing it a priori? If I then need to choose just one Q value I may choose, e.g. the more conservative one that still gives me some true discovery or the one that gives the maximum number of true discovery, etc.

  2. Am I doing again multiple comparisons in checking different levels of Q? I don't think so since I'm actually not testing new hypothesis but just controlling the FDR to different levels: the original hypothesis keep untouched, what is changed are the criteria in the BH procedure.

$\endgroup$
7
$\begingroup$

Yes, these things are done quite often, at least in genetics. To address your specific points:

  1. This is quite commonly done, and I have personally reported results this way. Though, make sure to make it clear at what level of FDR you are reporting. Think of it akin to "marginal" significance; people may find it interesting, but they have to know what they're looking at.

  2. No, you would not have to do multiple corrections for this, as you mentioned you're simply manipulating the $P$-values in a different way and not changing anything.

I would also look further into what Dr. Motulsky has mentioned above, if I were you. Reporting the $q$ value is a very common and useful metric.

$\endgroup$
  • $\begingroup$ Chirs could you share with me the reference were you have done this procedure? Thanks!!! $\endgroup$ – marina Aug 10 '15 at 8:41
  • 1
    $\begingroup$ Hi Marina, this is the reference that I was talking about (that's me as first author). In table 3 and 4, I reported $P$-values for a given FDR, noting the FDR in the footnotes of the table. These results were really marginally significant, so we really wanted to put them just in case they were replicated somewhere else. We weren't claiming that they were genome-wide significant at all, just that the results were something people might want to look into further. If you have any questions, I'm on quite often. Cheers $\endgroup$ – Chris C Aug 10 '15 at 12:00
5
$\begingroup$

An alternative you might find useful is to report -- for each comparison -- the q value. The q value (lower case) is the Q value (upper case) at which that particular comparison would be right at the border of being a discovery. You can then report the q value for each comparison, rather than just a list of which comparisons are "discoveries" using an arbitrary value of Q.

$\endgroup$
  • $\begingroup$ thanks for your suggestion. I think it goes in the same direction than Storey 2003 PNAS, but I still would like to clarify these two questions.thanks! $\endgroup$ – marina Aug 8 '15 at 18:04
0
$\begingroup$

There's nothing magical about alpha=.05.

I see nothing wrong with going with alpha/q=.10. I would also report confidence intervals (and adjust these as well).

Alternatively, use a Bayesian model with priors up to the job of wacking down false positives (horseshoe, laplace)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.