3
$\begingroup$

I am presenting an analysis in which I performed a number of tests for N variables. The variables which are significant at q < 0.05 are presented in a table (q is the Benjamini-Hochberg corrected p-value) along their q-value.

A colleague argues that since we are only presenting a few variables, one should show the actual p-values rather than the q-values in the table, since without the context (all other variables that were not significant) the context is lost and the q-values are misleading.

What do you think?

$\endgroup$
6
  • $\begingroup$ Would presenting both help? In the filed in which I work (health) the journal might well encourage you to put all the results including the ones which failed to reach some arbitrary value into supplementary material. $\endgroup$
    – mdewey
    Commented Dec 15, 2017 at 14:51
  • $\begingroup$ along their q value - what does it mean? $\endgroup$
    – user10619
    Commented Dec 15, 2017 at 16:53
  • $\begingroup$ I am completely lost as to how anybody sees similarity between the two questions 8-o $\endgroup$
    – January
    Commented Dec 16, 2017 at 11:42
  • $\begingroup$ @mdewey: unfortunately, this is not an option. $\endgroup$
    – January
    Commented Dec 16, 2017 at 11:43
  • 3
    $\begingroup$ I am voting to reopen this question as the proposed duplicate was asked by someone trying to understand the technique. In this case I believe the OP knows the difference between q and p but is asking for advice about presentation. $\endgroup$
    – mdewey
    Commented Dec 16, 2017 at 15:00

2 Answers 2

2
$\begingroup$

If there is a tradition in your field of presenting one or the other I would suggest presenting the $q$ values as these are the ones which are leading to the decision about statistical significance. In my experience reviewing articles you need to be careful to explain in the table caption exactly what they are to avoid ambiguity and forestall possible questions from reviewers who do not understand the technique.

I am a bit concerned about the fact that the other variables are not going to be reported as this poses problems for subsequent meta-analysts and also fails to give the community the full benefit of your work. But if you work in a field where people routinely do not publish such things then it will pass muster.

$\endgroup$
2
  • $\begingroup$ +1. However, regarding your concern: if e.g. variables are genes, then N is around 20000 and reporting all p-values is a bit problematic due to space limitations. $\endgroup$
    – amoeba
    Commented Dec 18, 2017 at 14:58
  • $\begingroup$ @amoeba: precisely so; however, we still report everything (and their dog) in the supplementary materials. $\endgroup$
    – January
    Commented Jan 23, 2018 at 10:48
1
$\begingroup$

Your colleague has it exactly backwards, in my opinion. (Especially) without context, presenting only the uncorrected p-values would be the misleading option of the two, since these values overestimate the odds against your findings under the null hypothesis. And without context, people have no way of knowing how great this exaggeration is.

For example, suppose you did 1000 tests. The Bonferroni-corrected significance threshold for this population of tests is $p < \frac{0.05}{1000}$. You find a comparison that just meets this threshold, so its Bonferroni-adjusted p-value is just under 0.05. This means that your odds of finding at such a result, among all your tests, were about 1 in 20. If you instead report the uncorrected p-value of $p<5\times10^{-5}$, your audience may wrongly conclude that the odds against your finding were about 1 in 20,000.

So it's precisely because you've lost the context of the other tests that the raw p-values would be (extra) misleading, because they cannot be fairly interpreted when reported separately from the other comparisons that you did. So it now falls on you to compensate for this loss of context by correcting your reported p-values for the effect of your multiple comparisons. In addition to which, of course, you should also report the number of those comparisons and the correction procedure.

In practice you nearly always single out certain tests above others, since the whole point of statistical analyses is to identify interesting effects, so you should always report p-values corrected for multiple comparisons (even if you do report the outcomes of the other tests somewhere).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.