There is convention concerning the use of asterisks to represent levels of significance. I am sure there are others, but let's stick to this one for now: Convention for symbols indicating statistical significance?
* P ≤ 0.05
** P ≤ 0.01
*** P ≤ 0.001
**** P ≤ 0.0001
I usually read this as
* There is a 1-in-20 chance that randomness (incorrectly) confirmed this hypothesis.
** There is a 1-in-100 chance ...
*** There is a 1-in-1000 chance ...
**** There is a 1-in-10,000 chance ...
Now, irrespective of the above (or not?), we use a threshold for statistical significance, which very often coincides with the thresholds for a single asterisk, *
. With multiple testing, we usually divide this significance threshold by the number of comparisons. E.g., when doing 40 comparisons, we use a value of 0.05 / 40 = 0.00125
: a result with p = 0.03
would be considered significant if we did not do a Bonferroni correction (0.03 < 0.05
), but it would not be with Bonferroni correction (0.03 > 0.00125
).
My question is: does one apply this Bonferroni correction to the cut-off values for asterisks as well? With 40 comparisons, will I use the above convention or the following one:
* P ≤ 0.00125
** P ≤ 0.00025
*** P ≤ 0.000025
**** P ≤ 0.0000025
For example, will a result with p = 0.0011
have two asterisks (according to the first convention), or just a single asterisk (according to the second convention)?
My gut feeling is that this depends on the presentation of the result. The result "one of the 40 hypotheses is true" should only receive a single asterisk, because the my interpretation is that there is a 1-in-20 chance that randomness (incorrectly) confirmed at least one of the hypotheses. On the other hand, when all 40 comparisons are presented as 40 individual hypotheses, the odds for each specific one to be confirmed incorrectly by randomness is still 1 in 800, and it should be presented with two asterisks. Then, the reader can decide how they aggregate these individual results into one. For example, if only one out of 40 hypothesis is **
according to the uncorrect, first convention, the reader might conclude themselves that overall, this is only *
in aggregate. As another extreme, having 40 individual **
according to the uncorrected, first convention might well aggregate as **
.
(Note that there is similar question in Do asterisks representing significance in figures refer to the p-value or to the significance level?, but without an answer that would answer this one here.)