Is it wrong to refer to results as "nearly" or "somewhat" significant?

Question

The general consensus on a similar question, Is it wrong to refer to results as being "highly significant"? is that "highly significant" is a valid, though non-specific, way to describe the strength of an association that has a p-value far below your pre-set significance threshold. However, what about describing p-values that are slightly above your threshold? I have seen some papers use terms like "somewhat significant", "nearly significant", "approaching significance", and so on. I find these terms to be a little wishy-washy, in some cases a borderline disingenuous way to pull a meaningful result out of a study with negative results. Are these terms acceptable to describe results that "just miss" your p-value cutoff?

Answer 1

If you want to allow "significance" to admit of degrees then fair enough ("somewhat significant", "fairly significant"), but avoid phrases that suggest you're still wedded to the idea of a threshold, such as "nearly significant", "approaching significance", or "at the cusp of significance" (my favourite from "Still Not Significant" on the blog Probable Error), if you don't want to appear desperate.

Answer 2

From my perspective, the issue boils down to what it actually means to carry out a significance test. Significance testing was devised as a means of making the decision of either to reject the null hypothesis or to fail to reject it. Fisher himself introduced the infamous 0.05 rule for making that (arbitrary) decision.

Basically, the logic of significance testing is that the user has to specify an alpha level for rejecting the null hypothesis (conventionally 0.05) before collecting the data. After completing the significance test, the user rejects the null if the p value is smaller than the alpha level (or fails to reject it otherwise).

The reason why you cannot declare an effect to be highly significant (say, at the 0.001 level) is because you cannot find stronger evidence than you set out to find. So, if you set your alpha level at 0.05 before the test, you can only find evidence at the 0.05 level, regardless of how small your p values is. In the same way, speaking of effects that are "somewhat significant" or "approaching significance" also doesn't make much sense because you chose this arbitrary criterion of 0.05. If you interpret the logic of significance testing very literally, anything bigger than 0.05 is not significant.

I agree that terms like "approaching significance" are often used to enhance the prospects of publication. However, I do not think that authors can be blamed for that because the current publication culture in some sciences still heavily relies on the "holy grail" of 0.05.

Some of these issues are discussed in:

Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.

Royall, R. (1997). Statistical evidence: a likelihood paradigm (Vol. 71). CRC press.

Answer 3

This slippery slope calls back to the Fisher vs Neyman/Pearson framework for null-hypothesis significance testing (NHST). On the one hand, one wants to make a quantitative assessment of just how unlikely a result is under the null hypothesis (e.g., effect sizes). On the other hand, at the end of the day you want a discrete decision as to whether your results are, or are not, likely to have been due to chance alone. What we've ended up with is a kind of hybrid approach that isn't very satisfying.

In most disciplines, the conventional p for significance is set at 0.05, but there is really no grounding for why this must be so. When I review a paper, I have absolutely no problem with an author calling 0.06 significant, or even 0.07, provided that the methodology is sound, and the entire picture, including all analyses, figures, etc. tell a consistent and believable story. Where you run into problems is when authors attempt to make a story out of trivial data with small effect sizes. Conversely, I might not fully 'believe' a test is practically meaningful even when it reaches conventional p < 0.05 significance. A colleague of mine once said: "Your statistics should simply back up what is already apparent in your figures."

That all said, I think Vasilev is correct. Given the broken publication system, you pretty much have to include p values, and therefore you pretty much have to use the word 'significant' to be taken seriously, even if it requires adjectives like "marginally" (which I prefer). You can always fight it out in peer review, but you have to get there first.

Answer 4

The difference between two p-values itself typically is not significant. So, it doesn't matter whether your p-value is 0.05, 0.049, 0.051...

With regards to p-values as a measure of strength of association: A p-value is not directly a measure of strength of association. A p-value is the probability of finding as extreme or more extreme data as the data you have observed, given the parameter is hypothesized to be 0 (if one's interested in the null hypothesis -- see Nick Cox' comment). However, this is often not the quantity the researcher is interested in. Many researchers are rather interested in answering questions like "what's the probability of the parameter to be greater than some chosen cut-off value?" If this is what you're interested in, you need to incorporate additional prior information in your model.

Answer 5

Whether "nearly significant" makes sense or not depends on one's philosophy of statistical inference. It's perfectly valid to consider the alpha level as a line in the sand, in which case one should only pay attention to whether $p<\alpha$ or $p>\alpha$. For such an "absolutist", "nearly significant" makes no sense. But it's also perfectly valid to think of p values as providing continuous measures of strength of support (not strength of effect, of course). For such a "continualist", "nearly significant" is a sensible way to describe a result with a moderate p-value. The problem arises when people mix these two philosophies - or worse, are not aware that both exist. (By the way - people often assume these map cleanly onto Neyman/Pearson and Fisher, but they don't; hence my admittedly clumsy terms for them). More detail about this in a blog post on this subject here: https://scientistseessquirrel.wordpress.com/2015/11/16/is-nearly-significant-ridiculous/

Answer 6

I tend to think saying something is almost statistically significant is not correct from a technical standpoint. Once you set your tolerance level the statistical test of significance is set. You have to go back to the idea of sampling distributions. If your tolerance level is say 0.05 and you happen to get a p-value of 0.053 then it is just by chance that the sample used yielded that statistic. You could very well get another sample that may not yield the same results- I believe the probability of that occurring is based on the tolerance level set and not on the sample statistic. Remember that you are testing samples against a population parameter and samples have their own sampling distribution. So in my opinion, either something is statistically significant or it is not.

Answer 7

The p-value is uniformly distributed on $[0,1]$ under $\mathcal{H}_0$ so getting a result with a p-value of 0.051 is as likely as getting a result with a p-value of 1. Since you have to set the significance level before getting data you reject the null for every p-value $p > \alpha$. Since you don't reject your null, you have to assume a uniformly distributed p-value, a higher or lower value is essentially meaningless.

This is a wholly different story when you reject the null, since the p-value is not uniformly distributed under $\mathcal{H}_1$ but the distribution depends on the parameter.

See for example Wikipedia.