7
$\begingroup$

I have done 350 independent tests. I need to correct them for the multiple comparisons. I am using the Benjamini-Hochberg (BH) procedure to control FDR. However choosing the Q (the level at which we want to control the FDR) beforehand could be restrictive (say e.g. I find no discovery for Q=0.05 and 40 for Q=0.1: it would be nice tell something about those 36 supposedly true discoveries made with Q=0.1 instead of reporting no discovery).

I have two questions:

  1. If I'm doing an exploratory analysis, is it a wrong procedure to have a look at the number of discoveries for varying levels of Q instead of fixing it a priori? If I then need to choose just one Q value I may choose, e.g. the more conservative one that still gives me some true discovery or the one that gives the maximum number of true discovery, etc.

  2. Am I doing again multiple comparisons in checking different levels of Q? I don't think so since I'm actually not testing new hypothesis but just controlling the FDR to different levels: the original hypothesis keep untouched, what is changed are the criteria in the BH procedure.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
7
$\begingroup$

Yes, these things are done quite often, at least in genetics. To address your specific points:

  1. This is quite commonly done, and I have personally reported results this way. Though, make sure to make it clear at what level of FDR you are reporting. Think of it akin to "marginal" significance; people may find it interesting, but they have to know what they're looking at.

  2. No, you would not have to do multiple corrections for this, as you mentioned you're simply manipulating the $P$-values in a different way and not changing anything.

I would also look further into what Dr. Motulsky has mentioned above, if I were you. Reporting the $q$ value is a very common and useful metric.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Chirs could you share with me the reference were you have done this procedure? Thanks!!! $\endgroup$
    – marina
    Commented Aug 10, 2015 at 8:41
  • 1
    $\begingroup$ Hi Marina, this is the reference that I was talking about (that's me as first author). In table 3 and 4, I reported $P$-values for a given FDR, noting the FDR in the footnotes of the table. These results were really marginally significant, so we really wanted to put them just in case they were replicated somewhere else. We weren't claiming that they were genome-wide significant at all, just that the results were something people might want to look into further. If you have any questions, I'm on quite often. Cheers $\endgroup$
    – Chris C
    Commented Aug 10, 2015 at 12:00
5
$\begingroup$

An alternative you might find useful is to report -- for each comparison -- the q value. The q value (lower case) is the Q value (upper case) at which that particular comparison would be right at the border of being a discovery. You can then report the q value for each comparison, rather than just a list of which comparisons are "discoveries" using an arbitrary value of Q.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thanks for your suggestion. I think it goes in the same direction than Storey 2003 PNAS, but I still would like to clarify these two questions.thanks! $\endgroup$
    – marina
    Commented Aug 8, 2015 at 18:04
1
$\begingroup$

There's nothing magical about alpha=.05.

I see nothing wrong with going with alpha/q=.10. I would also report confidence intervals (and adjust these as well).

Alternatively, use a Bayesian model with priors up to the job of wacking down false positives (horseshoe, laplace)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.