13
$\begingroup$

I sometimes speak about statistical results to a popular audience, and the term "significant" can (understandably) be misunderstood. I sometimes want to say something like "the likelihood of seeing these results under the null hypothesis is small enough that there probably is something going on here, however the effect size is small enough that it's probably not worth worrying too much about."

Among technical audiences, I believe this could be summarized as "small but significant". Is there a short way I can communicate this to a non-technical audience?

$\endgroup$
  • 16
    $\begingroup$ "Statistically detectable" is one of my favourites to get rid of the misleading term "significant", especially when talking to non-statisticians. $\endgroup$ – Michael M Jun 25 at 17:03
  • 2
    $\begingroup$ @MichaelM I think you should expand on that and post as an answer. I like it a lot, too, and I am going to use it! $\endgroup$ – Dave Jun 25 at 17:05
  • 2
    $\begingroup$ I also like "statistically detectable", especially for a lay audience. But you asked how to compactly communicate two things: that, plus that it's not important. So I'd suggest (words separated by slashes are meant to represent different options): "statistically detectable, but not biologically/physically/economically [insert your field here] meaningful/important/significant". $\endgroup$ – theorist Jun 26 at 1:26
  • 2
    $\begingroup$ I think this question would be clearer if you wrote "significant but small," since what you are emphasizing is that the difference is negligible. $\endgroup$ – Max Jun 26 at 4:18
  • 2
    $\begingroup$ @theorist: +1, I believe it captures the essence succinctly. Alternatively: "statistically detectable, but of no practical relevance". $\endgroup$ – Igor F. Jun 30 at 7:27
14
$\begingroup$

If the audience knows what "statistical significant" and $p \le 0.05$ means, there is not much that can go wrong. But otherwise, I really like the fantastic suggestion by Jordan Ellenberg as an alternative to "statistically significant" in general:

[...] "statistically noticeable” or statistically detectable" instead of “statistically significant”! That would be truer to the meaning of the method [...] - Jordan Ellenberg in his book "How Not to Be Wrong: The Power of Mathematical Thinking"

Edit based on the short discussion in the comments: Note this answer does not specifically address the "small effect" situation but rather proposes an apt word for statistical significance in general where you don't have to fear that people take "statistically significant" as "relevant". In this way, you can distinguish in a clean and understandable way the two topics "hypothesis testing" and "effect size".

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ -1, as this does not address the effect size issue in any way. With enough data, you might show that a new drug lowers cholesterol in a statistically significant manner by 0.000001%, which is a likely clinically meaningless finding. Simply stating that a difference is "statistically significant" gives no indication whatsoever whether the difference is meaningful or not. The headline "New Drug Statistically Significantly Lowers Cholesterol", while true, omits the more important part of the story, which is that the drug will have near-zero effect on human health. $\endgroup$ – Nuclear Wang Jun 26 at 14:22
  • 3
    $\begingroup$ This is correct and originally a reason to post the term "detectable" only within a comment. Still, instead of using a misleading term to stress significance and then relativise its meaning for small effects, a strategy is to use an apt word from the beginning and then maybe throw in the fact that the effect is large/small (which is an aspect not part of the concept of testing). $\endgroup$ – Michael M Jun 26 at 15:00
10
$\begingroup$

I have found the distinction between statistically significant and physically significant is often useful when communicating results to non-statisticians. The phrases clinically significant or practically significant may be preferred in some fields (thanks @Jelsema). You are describing a situation where the effect may be statistically significant, but physically insignificant.


As an aside, there is actually a push right now to stop (or limit) using the phrase "statistically significant" altogether. Some relevant reading:

Moving to a World Beyond "p < 0.05"

The Difference Between "Significant" and "Not Significant" is Not Itself Statistically Significant

Scientists rise up against statistical significance

References

Wasserstein, Ronald L., Allen L. Schirm, and Nicole A. Lazar. "Moving to a world beyond “p< 0.05”." (2019): 1-19.

Gelman, Andrew, and Hal Stern. "The difference between “significant” and “not significant” is not itself statistically significant." The American Statistician 60.4 (2006): 328-331.

Amrhein, Valentin, Sander Greenland, and Blake McShane. "Scientists rise up against statistical significance." (2019): 305-307.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ Just to tack on to the language aspect, with medical researchers I've used "clinically" significant to highlight the different between statistical results and practical meaning. $\endgroup$ – Jelsema Jun 25 at 17:27
  • $\begingroup$ I much prefer "practically" (as opposed to "physically") here, to imply it's significant in practice (as opposed to in theory). It's also synonymous with "basically", which would also work if it's insignificant, but less so if it's significant. $\endgroup$ – Bernhard Barker Jun 26 at 10:47
  • $\begingroup$ @Bernhard, thanks for the perspective! I often use "physically" because I work with a lot of engineers, but I understand that it may not be the best word-choice for most disciplines. $\endgroup$ – knrumsey Jun 26 at 16:04
  • 1
    $\begingroup$ "practically significant" works well enough across many fields. But the bigger idea is to use the adverb that registers with the readership or audience. See also "economically", as in "the error is millions of dollars, but that is not economically significant". $\endgroup$ – Nick Cox Jun 30 at 7:16
3
$\begingroup$

I would use something like, "We notice a difference that is extremely likely to be there and not a fluke. However, the difference has no practical meaning. They are effectively the same, like the distance to the sun versus the distance to the sun plus an inch."

The acknowledges the statistical significance ("not a fluke") while acknowledging the lack of practical significance ("no practical meaning").

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Referring to the tale of "The Princess and the Pea", you could say :

"There seems to be a pea under all these mattresses."

See also: https://en.wikipedia.org/wiki/The_Princess_and_the_Pea

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ But the pea mattered to the princess... $\endgroup$ – Dave Jun 27 at 19:05
  • $\begingroup$ There are princesses who can detect peas under different beds in parallel universes, $\endgroup$ – Nick Cox Jun 30 at 7:17
  • $\begingroup$ @Dave: Yes, and the statistician is here the princess, in the sense that the statistician was able to detect it. To all other folks in the world, the pea doesn't matter, because they cannot detect it. $\endgroup$ – Igor F. Jun 30 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.