20
$\begingroup$

In a Wilcoxon signed-ranks statistical significance test, we came across some data that produces a $p$-value of $0.04993$. With a threshold of $p < 0.05$, is this result enough to reject the null hypothesis, or is it safer to say the test was inconclusive, since if we round the p-value to 3 decimal places it becomes $0.050$?

$\endgroup$
  • 24
    $\begingroup$ 0.04993 < 0.05, so it's just lower. Your instinct is good that no P-value can be trusted to several decimal places, but if the program says less than 0.05, people generally take it as delivered. The real issue here is making a fetish of fixed-level significance testing so that < 0.05 means "real", "publishable", "cause for happiness" and the opposite means "illusory", "not publishable", "cause for misery". Most good introductory texts on statistics discuss this to some extent. One good one is Freedman, Pisani, Purves, Statistics. New York: W.W. Norton, any edition. $\endgroup$ – Nick Cox Jun 4 '13 at 9:35
  • 8
    $\begingroup$ You have to ask yourself what would be your decision if the p-value is 0.051? what if it is 0.049? Would you make different decisions? Why? $\endgroup$ – AlefSin Jun 4 '13 at 9:43
  • 2
    $\begingroup$ Thank you for your comments. In our case we are not pondering whether the data is publishable or not, etc... We are simply considering making a statement in the paper about the statistical significance of this result, and we want to make sure our statement is not incorrect or inaccurate. $\endgroup$ – Islam El-Nabarawy Jun 4 '13 at 10:52
  • 3
    $\begingroup$ Reporting P=0.04993 is what springs to mind. It's difficult to predict reviewers' or editors' comments. If you want to round, specifying a consistent rounding convention is always a good idea and widely acceptable. Some people would round to 3 d.p. and might also use some kind of starring convention so reporting 0.050 (3 d.p.) and starring it as <0.05 are consistent. $\endgroup$ – Nick Cox Jun 4 '13 at 11:27
  • 2
    $\begingroup$ I dunno... maybe we should run a double bootstrap and calculate a confidence interval for the $p$-value! In all honesty, I would report: "The findings were borderline significant, $0.049 < p < 0.050$." At that point, you're splitting hairs, and everyone suddenly remembers that 1/20 odds of a false positive is a completely arbitrary way to run science. $\endgroup$ – AdamO Oct 11 '13 at 22:56
21
$\begingroup$

There are two issues here:

1) If you're doing a formal hypothesis test (and if you're going as far as quoting a p-value in my book you already are), what is the formal rejection rule?

When comparing test statistics to critical values, the critical value is in the rejection region. While this formality doesn't matter much when everything is continuous, it does matter when the distribution of the test statistic is discrete.

Correspondingly, when comparing p-values and significance levels, the rule is:

          Reject if $p\leq\alpha$

Please note that, even if you rounded your p-value up to 0.05, indeed even if the $p$ value was exactly 0.05, formally, you should still reject.

2) In terms of 'what is our p-value telling us', then assuming you can even interpret a p-value as 'evidence against the null' (let's say that opinion on that is somewhat divided), 0.0499 and 0.0501 are not really saying different things about the data (effect sizes would tend to be almost identical).

My suggestion would be to (1) formally reject the null, and perhaps point out that even if it were exactly 0.05 it should still be rejected; (2) note that there's nothing particularly special about $\alpha = 0.05$ and it's very close to that borderline -- even a slightly smaller significance threshold would not lead to rejection.

$\endgroup$
  • 2
    $\begingroup$ But, again, you can use very similar arguments to not reject null. There is nothing special about 0.05, if you had chosen 0.06 as your limit you would probably not be asking the question, but the situation would not be that much different... Rather in these situations I would ask: "what is the real-life meaning of this result?". For instance if this was a biological experiment I would look for the biological significance of the specific result, report the p-value as it is and rather comment on the biology. $\endgroup$ – nico Oct 12 '13 at 22:15
  • $\begingroup$ @nico this was already the point of my item (2); it argues against over-reliance on the formal approach in (1) $\endgroup$ – Glen_b Oct 12 '13 at 22:18
  • $\begingroup$ Thank you Glen and nico. This part of the data was secondary to our experiments, so we just ended up reporting the value as is. In any case, I am marking this as the accepted answer. Thanks again to everyone who participated with answers or comments. $\endgroup$ – Islam El-Nabarawy Oct 13 '13 at 0:32
5
$\begingroup$

It lies in the eye of the beholder.

Formally, if there is a strict decision rule for your problem, follow it. This means $\alpha$ is given. However, I am not aware of any problem where this is the case (though setting $\alpha=0.05$ is what many practitioners do after Statistics101).

So it really boils down to what AlefSin commented before. There cannot be a "correct answer" to your question. Report what you got, rounded or not.

There is a huge literature on the "significance of significance"; see for example the recent paper of one of the leading German statisticians Walter Krämer on "The cult of statistical significance - What economists should and should not do to make their data talk", Schmollers Jahrbuch 131, 455-468, 2011.

$\endgroup$
-6
$\begingroup$

The answer is absolutely not. There is no "in the eye of the beholder", there is no argument, the answer is no, your data is not significant at the $p=0.05$ level. (Ok, there is one way out, but its a very narrow path.)

The key problem is this phrase: "We came across some data...".

This suggests that you looked at several other statistical hypothesis, and rejected them because they did not reach your significance level. You found one hypothesis that (barely) met your standard, and you are wondering whether it is significant. Unless your $p$ value accounts for such multiple hypothesis testing, it is overly optimistic. Given that you are just three decimal points away from your threshold, considering even one additional hypothesis would surely push $p$ over the line.

There is a name for this sort of statistical malfeasance: data dredging. I'm ambivalent about reporting it in the paper as an interesting hypothesis; does it have some physical reason you expect it to hold?

There is, however, one way out. Perhaps you decided a priori to perform just this one test on just this one data set. You wrote that down in your lab notebook, in front of someone so that you could prove it later. Then you did your test.

If you did this, then your result is valid at the $p=0.05$ level, and you can back it up to skeptics like me. Otherwise, sorry, it is not a statistically significant result.

$\endgroup$
  • 14
    $\begingroup$ This may be over-relying on a particular choice of phrasing; you're assuming rather a lot from what might be simply a poor choice of words - not everyone here has English as a first language. It's definitely worth raising as a potential problem, but to simply state things so baldly ("absolutely not") implies you know more than we can tell from what's here. (Further, the reference to a 'lab notebook' implies the OP is doing work in a lab. I doubt this is the case. Again, you imply you know more than we have here.) $\endgroup$ – Glen_b Oct 12 '13 at 20:56
  • 10
    $\begingroup$ Mike McCoy, thank you for your answer, but I'm afraid in this case Glen_b is correct. I am not a native English speaker, and while I strive to write and speak as fluently as my skills allow, usage and connotation continue to elude me. So, in this particular case, we didn't try different things until we found something that was significant. Actually, what we were trying to prove is that there were no statistically significant increase in some error value, and in one particular case we found that the error was actually reduced, and when we ran the W test, this is where we got the 0.0499. $\endgroup$ – Islam El-Nabarawy Oct 13 '13 at 0:25
  • 1
    $\begingroup$ Mike, I also did not see a problem in the phrasing of the question. And it seems nobody else saw signs of data snooping, mining, dredging, whatsoever here ... And it definitely lies in the eye of the beholder. There is no mathematical fact but a decision rule chosen by the statistician. Re-read what AlefSin, Glen in his point (2) and I wrote. $\endgroup$ – Skullduggery Oct 13 '13 at 6:56
  • 3
    $\begingroup$ @IslamEl-Nabarawy If you wanted to establish equivalence/lack of difference, you have many other problems than how to interpret a value close to the threshold or potential data snooping. Just finding a p-value slightly over .05 (or whatever error level you choose) is definitely not enough. Look up “testing for equivalence” here and elsewhere or ask a question specifically about that because it's an entirely different problem. $\endgroup$ – Gala Oct 13 '13 at 10:32
  • 4
    $\begingroup$ "There is, however, one way out. Perhaps you decided a priori to perform just this one test on just this one data set. You wrote that down in your lab notebook, in front of someone so that you could prove it later. Then you did your test. If you did this, then your result is valid at the p=0.05 level, and you can back it up to skeptics like me. Otherwise, sorry, it is not a statistically significant result" Talk about guilty until proven innocent. So, in the absence of forensic evidence ruling out academic dishonesty, an analysis is worthless? Sheesh. $\endgroup$ – GoF_Logistic Mar 3 '17 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.