0
$\begingroup$

There is convention concerning the use of asterisks to represent levels of significance. I am sure there are others, but let's stick to this one for now: Convention for symbols indicating statistical significance?

*       P ≤ 0.05
**      P ≤ 0.01
***     P ≤ 0.001
****    P ≤ 0.0001

I usually read this as

*      There is a 1-in-20 chance that randomness (incorrectly) confirmed this hypothesis.
**     There is a 1-in-100 chance ...
***    There is a 1-in-1000 chance ...
****   There is a 1-in-10,000 chance ...

Now, irrespective of the above (or not?), we use a threshold for statistical significance, which very often coincides with the thresholds for a single asterisk, *. With multiple testing, we usually divide this significance threshold by the number of comparisons. E.g., when doing 40 comparisons, we use a value of 0.05 / 40 = 0.00125: a result with p = 0.03 would be considered significant if we did not do a Bonferroni correction (0.03 < 0.05), but it would not be with Bonferroni correction (0.03 > 0.00125).

My question is: does one apply this Bonferroni correction to the cut-off values for asterisks as well? With 40 comparisons, will I use the above convention or the following one:

*       P ≤ 0.00125
**      P ≤ 0.00025
***     P ≤ 0.000025
****    P ≤ 0.0000025

For example, will a result with p = 0.0011 have two asterisks (according to the first convention), or just a single asterisk (according to the second convention)?

My gut feeling is that this depends on the presentation of the result. The result "one of the 40 hypotheses is true" should only receive a single asterisk, because the my interpretation is that there is a 1-in-20 chance that randomness (incorrectly) confirmed at least one of the hypotheses. On the other hand, when all 40 comparisons are presented as 40 individual hypotheses, the odds for each specific one to be confirmed incorrectly by randomness is still 1 in 800, and it should be presented with two asterisks. Then, the reader can decide how they aggregate these individual results into one. For example, if only one out of 40 hypothesis is ** according to the uncorrect, first convention, the reader might conclude themselves that overall, this is only * in aggregate. As another extreme, having 40 individual ** according to the uncorrected, first convention might well aggregate as **.

(Note that there is similar question in Do asterisks representing significance in figures refer to the p-value or to the significance level?, but without an answer that would answer this one here.)

$\endgroup$
  • $\begingroup$ There is a 1-in-20 chance that randomness (incorrectly) confirmed this hypothesis. - This is not a correct interpretation of a $p$-value. Moreover, The result "one of the 40 hypotheses is true" should only receive a single asterisk is not correct either. There are multiple issues with your interpretation of $p$-values: (1) You do not prove hypotheses with them; (2) no procedure of significance testing / multiple testing correction / significance indication will guarantee any number of hypotheses to be true or false, these are still subject to chance; ... $\endgroup$ – Frans Rodenburg Aug 4 '18 at 2:35
  • $\begingroup$ ... (3) how nominal significance translates to the FWER or FDR is dependent on the number of tests and (in the case of FDR) the number of nominally significant tests, so you cannot say how the nominal stars should be interpreted without seeing the actual $p$-values and the total number of tests. $\endgroup$ – Frans Rodenburg Aug 4 '18 at 2:38
1
$\begingroup$

IMHO, any problem involving multiple comparisons should display actual p-values anyway, so that the reader could do his own adjustments if he treats the hypotheses differently than the author. (It is actually a requirement in some journals, e.g. GENETICS.)
I am sure you could still use the asterisks to indicate the most significant findings, but the threshold is up to you. To take an example from large-scale genetics again, it is normal to have thousands of markers with $p<0.01$, so any significance marks in such studies use adjusted thresholds.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.