The general consensus on a similar question, Is it wrong to refer to results as being "highly significant"? is that "highly significant" is a valid, though non-specific, way to describe the strength of an association that has a p-value far below your pre-set significance threshold. However, what about describing p-values that are slightly above your threshold? I have seen some papers use terms like "somewhat significant", "nearly significant", "approaching significance", and so on. I find these terms to be a little wishy-washy, in some cases a borderline disingenuous way to pull a meaningful result out of a study with negative results. Are these terms acceptable to describe results that "just miss" your p-value cutoff?
$\begingroup$
$\endgroup$
6
asked Sep 17, 2015 at 14:15
-
3$\begingroup$ I don't believe anyone suggested qualifying "significance" to describe the "strength of an association"; the latter sounds more like a measure of effect size. Anyway, see here for a fuller list. $\endgroup$– Scortchi - Reinstate Monica ♦Sep 17, 2015 at 14:21
-
1$\begingroup$ @Scortchi - From my understanding, a very small p-value is highly significant, meaning a strong association between the variable in question and the target. This is the result of a large effect size, a lot of data, or both. For large p-values, the evidence supporting an association between variable and target is weak. Also, love that list in your link. $\endgroup$– Nuclear HoagieSep 17, 2015 at 14:37
-
9$\begingroup$ Obtaining a very small p-value for a small effect size could scarcely be called a "strong association." It would only be a detectable association. $\endgroup$– whuber ♦Sep 17, 2015 at 14:48
-
2$\begingroup$ I've seen people using these phrases a lot in the industry, not in academic papers though. $\endgroup$– AksakalSep 17, 2015 at 19:10
-
1$\begingroup$ Perhaps your discomfort comes from believing that p-values (or any other number derived from a sample) are sharp measures of something. $\endgroup$– Eric TowersSep 18, 2015 at 4:51
|
Show 1 more comment
7 Answers
$\begingroup$
$\endgroup$
4
If you want to allow "significance" to admit of degrees then fair enough ("somewhat significant", "fairly significant"), but avoid phrases that suggest you're still wedded to the idea of a threshold, such as "nearly significant", "approaching significance", or "at the cusp of significance" (my favourite from "Still Not Significant" on the blog Probable Error), if you don't want to appear desperate.
answered Sep 17, 2015 at 15:42
Scortchi - Reinstate Monica♦Scortchi - Reinstate Monica
28.6k8 gold badges87 silver badges268 bronze badges
-
9$\begingroup$ (+1) for the link. But I think the highlight of poetic creativity there is "teetering on the brink of significance (p=0.06)". $\endgroup$ Sep 17, 2015 at 20:45
-
1$\begingroup$ @AlecosPapadopoulos: You're right, though "flirting with conventional levels of significance" & "hovering closer to statistical significance" deserve honourable mentions. "Quasi-significant" is perhaps a winner in a different category. $\endgroup$ Sep 18, 2015 at 9:59
-
4$\begingroup$ Indeed the first two are of true cinematographic spirit, the first from the film "Statistical Gigolo" (who else would flirt with a conventional level?), while the second from the film "Dying on the Tail", where we see the menacing vulture (p-value) hovering over the dying hero (statistical significance). $\endgroup$ Sep 18, 2015 at 14:01
-
1$\begingroup$ Personally, I'd abandon the word 'significant' in my phrasing and call p=0.06 'quite interesting'. Rightly, or wrongly, when I first encountered p-values within a Six Sigma course, the instructor suggested that for 0.05<=0.1 the right label was 'more data required' (based on an industrial setting where additional data points are hard to acquire, so completely different to any 'Big Data' scenario $\endgroup$ Aug 1, 2016 at 0:39
$\begingroup$
$\endgroup$
1
From my perspective, the issue boils down to what it actually means to carry out a significance test. Significance testing was devised as a means of making the decision of either to reject the null hypothesis or to fail to reject it. Fisher himself introduced the infamous 0.05 rule for making that (arbitrary) decision.
Basically, the logic of significance testing is that the user has to specify an alpha level for rejecting the null hypothesis (conventionally 0.05) before collecting the data. After completing the significance test, the user rejects the null if the p value is smaller than the alpha level (or fails to reject it otherwise).
The reason why you cannot declare an effect to be highly significant (say, at the 0.001 level) is because you cannot find stronger evidence than you set out to find. So, if you set your alpha level at 0.05 before the test, you can only find evidence at the 0.05 level, regardless of how small your p values is. In the same way, speaking of effects that are "somewhat significant" or "approaching significance" also doesn't make much sense because you chose this arbitrary criterion of 0.05. If you interpret the logic of significance testing very literally, anything bigger than 0.05 is not significant.
I agree that terms like "approaching significance" are often used to enhance the prospects of publication. However, I do not think that authors can be blamed for that because the current publication culture in some sciences still heavily relies on the "holy grail" of 0.05.
Some of these issues are discussed in:
Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.
Royall, R. (1997). Statistical evidence: a likelihood paradigm (Vol. 71). CRC press.
-
1$\begingroup$ You're mixing Fisherian philosophy of science with Neyman/Pearson's approach if you add an alpha-level to Fisher's significance testing. $\endgroup$ Oct 3, 2015 at 11:48
$\begingroup$
$\endgroup$
This slippery slope calls back to the Fisher vs Neyman/Pearson framework for null-hypothesis significance testing (NHST). On the one hand, one wants to make a quantitative assessment of just how unlikely a result is under the null hypothesis (e.g., effect sizes). On the other hand, at the end of the day you want a discrete decision as to whether your results are, or are not, likely to have been due to chance alone. What we've ended up with is a kind of hybrid approach that isn't very satisfying.
In most disciplines, the conventional p for significance is set at 0.05, but there is really no grounding for why this must be so. When I review a paper, I have absolutely no problem with an author calling 0.06 significant, or even 0.07, provided that the methodology is sound, and the entire picture, including all analyses, figures, etc. tell a consistent and believable story. Where you run into problems is when authors attempt to make a story out of trivial data with small effect sizes. Conversely, I might not fully 'believe' a test is practically meaningful even when it reaches conventional p < 0.05 significance. A colleague of mine once said: "Your statistics should simply back up what is already apparent in your figures."
That all said, I think Vasilev is correct. Given the broken publication system, you pretty much have to include p values, and therefore you pretty much have to use the word 'significant' to be taken seriously, even if it requires adjectives like "marginally" (which I prefer). You can always fight it out in peer review, but you have to get there first.
$\begingroup$
$\endgroup$
2
The difference between two p-values itself typically is not significant. So, it doesn't matter whether your p-value is 0.05, 0.049, 0.051...
With regards to p-values as a measure of strength of association: A p-value is not directly a measure of strength of association. A p-value is the probability of finding as extreme or more extreme data as the data you have observed, given the parameter is hypothesized to be 0 (if one's interested in the null hypothesis -- see Nick Cox' comment). However, this is often not the quantity the researcher is interested in. Many researchers are rather interested in answering questions like "what's the probability of the parameter to be greater than some chosen cut-off value?" If this is what you're interested in, you need to incorporate additional prior information in your model.
-
6$\begingroup$ I agree with the spirit of this, but the small print as always needs total vigilance. "given the parameter is assumed to be 0": often, but not always. P-values can be calculated for other hypotheses too. Also, for "assumed" read "hypothesised". $\endgroup$– Nick CoxSep 17, 2015 at 15:45
-
$\begingroup$ You are totally right -- I'll edit my answer! $\endgroup$ Sep 21, 2015 at 6:16
$\begingroup$
$\endgroup$
Whether "nearly significant" makes sense or not depends on one's philosophy of statistical inference. It's perfectly valid to consider the alpha level as a line in the sand, in which case one should only pay attention to whether $p<\alpha$ or $p>\alpha$. For such an "absolutist", "nearly significant" makes no sense. But it's also perfectly valid to think of p values as providing continuous measures of strength of support (not strength of effect, of course). For such a "continualist", "nearly significant" is a sensible way to describe a result with a moderate p-value. The problem arises when people mix these two philosophies - or worse, are not aware that both exist. (By the way - people often assume these map cleanly onto Neyman/Pearson and Fisher, but they don't; hence my admittedly clumsy terms for them). More detail about this in a blog post on this subject here: https://scientistseessquirrel.wordpress.com/2015/11/16/is-nearly-significant-ridiculous/
$\begingroup$
$\endgroup$
I tend to think saying something is almost statistically significant is not correct from a technical standpoint. Once you set your tolerance level the statistical test of significance is set. You have to go back to the idea of sampling distributions. If your tolerance level is say 0.05 and you happen to get a p-value of 0.053 then it is just by chance that the sample used yielded that statistic. You could very well get another sample that may not yield the same results- I believe the probability of that occurring is based on the tolerance level set and not on the sample statistic. Remember that you are testing samples against a population parameter and samples have their own sampling distribution. So in my opinion, either something is statistically significant or it is not.
$\begingroup$
$\endgroup$
3
The p-value is uniformly distributed on $[0,1]$ under $\mathcal{H}_0$ so getting a result with a p-value of 0.051 is as likely as getting a result with a p-value of 1. Since you have to set the significance level before getting data you reject the null for every p-value $p > \alpha$. Since you don't reject your null, you have to assume a uniformly distributed p-value, a higher or lower value is essentially meaningless.
This is a wholly different story when you reject the null, since the p-value is not uniformly distributed under $\mathcal{H}_1$ but the distribution depends on the parameter.
See for example Wikipedia.
-
$\begingroup$ I don't quite follow you. Yes, in any continuous distribution, the likelihood of getting a result of exactly 0.051 is equal to the likelihood of getting a result of exactly 1 - it's zero. But hypothesis testing examines the likelihood of seeing a value at least as extreme as the one observed. You will always find a p-value at least as extreme as 1, but it's far less likely to see a p-value as extreme as 0.051. What makes that difference "meaningless"? $\endgroup$ Oct 31, 2018 at 12:22
-
$\begingroup$ Under the null it is as likely to observe a p-value in the interval [0.05,0.051] as it is to observe a p-value in the interval [0.999,1]. Observing a p-value closer to the threshold is not more evidence against the 0 as observing any other p-value outside the rejection area. $\endgroup$– snautOct 31, 2018 at 17:34
-
$\begingroup$ Some call a p value of 0.05 significant, others use 0.01 or 0.1 as a threshold. So, among 3 researchers that do the same analysis and find a p-value of 0.03, two might call it significant and one might not. If they all find a p-value of 0.91, none will call it significant. A p-value closer to the threshold means more individuals will deem there to be sufficient evidence to reject the null. I don't see why p=0.051 and p=1 should be indistinguishable in terms of support for H1 - some people will justifiably support H1 with p=0.051; nobody will do so with p=1. $\endgroup$ Oct 31, 2018 at 18:28