20
$\begingroup$

In a Wilcoxon signed-ranks statistical significance test, we came across some data that produces a $p$-value of $0.04993$. With a threshold of $p < 0.05$, is this result enough to reject the null hypothesis, or is it safer to say the test was inconclusive, since if we round the p-value to 3 decimal places it becomes $0.050$?

$\endgroup$
9
  • 27
    $\begingroup$ 0.04993 < 0.05, so it's just lower. Your instinct is good that no P-value can be trusted to several decimal places, but if the program says less than 0.05, people generally take it as delivered. The real issue here is making a fetish of fixed-level significance testing so that < 0.05 means "real", "publishable", "cause for happiness" and the opposite means "illusory", "not publishable", "cause for misery". Most good introductory texts on statistics discuss this to some extent. One good one is Freedman, Pisani, Purves, Statistics. New York: W.W. Norton, any edition. $\endgroup$
    – Nick Cox
    Jun 4, 2013 at 9:35
  • 9
    $\begingroup$ You have to ask yourself what would be your decision if the p-value is 0.051? what if it is 0.049? Would you make different decisions? Why? $\endgroup$
    – AlefSin
    Jun 4, 2013 at 9:43
  • 2
    $\begingroup$ Thank you for your comments. In our case we are not pondering whether the data is publishable or not, etc... We are simply considering making a statement in the paper about the statistical significance of this result, and we want to make sure our statement is not incorrect or inaccurate. $\endgroup$ Jun 4, 2013 at 10:52
  • 3
    $\begingroup$ Reporting P=0.04993 is what springs to mind. It's difficult to predict reviewers' or editors' comments. If you want to round, specifying a consistent rounding convention is always a good idea and widely acceptable. Some people would round to 3 d.p. and might also use some kind of starring convention so reporting 0.050 (3 d.p.) and starring it as <0.05 are consistent. $\endgroup$
    – Nick Cox
    Jun 4, 2013 at 11:27
  • 3
    $\begingroup$ I dunno... maybe we should run a double bootstrap and calculate a confidence interval for the $p$-value! In all honesty, I would report: "The findings were borderline significant, $0.049 < p < 0.050$." At that point, you're splitting hairs, and everyone suddenly remembers that 1/20 odds of a false positive is a completely arbitrary way to run science. $\endgroup$
    – AdamO
    Oct 11, 2013 at 22:56

5 Answers 5

21
$\begingroup$

There are two issues here:

1) If you're doing a formal hypothesis test (and if you're going as far as quoting a p-value in my book you already are), what is the formal rejection rule?

When comparing test statistics to critical values, the critical value is in the rejection region. While this formality doesn't matter much when everything is continuous, it does matter when the distribution of the test statistic is discrete.

Correspondingly, when comparing p-values and significance levels, the rule is:

          Reject if $p\leq\alpha$

Please note that, even if you rounded your p-value up to 0.05, indeed even if the $p$ value was exactly 0.05, formally, you should still reject.

2) In terms of 'what is our p-value telling us', then assuming you can even interpret a p-value as 'evidence against the null' (let's say that opinion on that is somewhat divided), 0.0499 and 0.0501 are not really saying different things about the data (effect sizes would tend to be almost identical).

My suggestion would be to (1) formally reject the null, and perhaps point out that even if it were exactly 0.05 it should still be rejected; (2) note that there's nothing particularly special about $\alpha = 0.05$ and it's very close to that borderline -- even a slightly smaller significance threshold would not lead to rejection.

$\endgroup$
3
  • 2
    $\begingroup$ But, again, you can use very similar arguments to not reject null. There is nothing special about 0.05, if you had chosen 0.06 as your limit you would probably not be asking the question, but the situation would not be that much different... Rather in these situations I would ask: "what is the real-life meaning of this result?". For instance if this was a biological experiment I would look for the biological significance of the specific result, report the p-value as it is and rather comment on the biology. $\endgroup$
    – nico
    Oct 12, 2013 at 22:15
  • $\begingroup$ @nico this was already the point of my item (2); it argues against over-reliance on the formal approach in (1) $\endgroup$
    – Glen_b
    Oct 12, 2013 at 22:18
  • $\begingroup$ Thank you Glen and nico. This part of the data was secondary to our experiments, so we just ended up reporting the value as is. In any case, I am marking this as the accepted answer. Thanks again to everyone who participated with answers or comments. $\endgroup$ Oct 13, 2013 at 0:32
5
$\begingroup$

It lies in the eye of the beholder.

Formally, if there is a strict decision rule for your problem, follow it. This means $\alpha$ is given. However, I am not aware of any problem where this is the case (though setting $\alpha=0.05$ is what many practitioners do after Statistics101).

So it really boils down to what AlefSin commented before. There cannot be a "correct answer" to your question. Report what you got, rounded or not.

There is a huge literature on the "significance of significance"; see for example the recent paper of one of the leading German statisticians Walter Krämer on "The cult of statistical significance - What economists should and should not do to make their data talk", Schmollers Jahrbuch 131, 455-468, 2011.

$\endgroup$
1
$\begingroup$

In light of the assumptions of your model, you should reject the null because dichotomizing claims based on hypothesis tests have clear epistemological and pragmatic functions. But never forget that: “No isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon; for the ‘one chance in a million’ will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us. In order to assert that a natural phenomenon is experimentally demonstrable we need, not an isolated record, but a reliable method of procedure. In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result.” Fisher, R. A. (1935). The design of experiments. Oliver & Boyd.

In the pragmatic sense, you should reject. In the statistical sense, you need more data.

$\endgroup$
1
$\begingroup$

The 0.05 threshold is a hurdle that you have set for yourself in order to enforce a degree of self-skepticism about your alternative hypothesis. It somewhat weakens that self-skepticism if you change the definition of the threshold after seeing the result. The real question is why you are performing an NHST, what do you think it tells you (probably not very much in most cases)?

The threshold should be set according to the nature of the experiment, so there is no one-true threshold anyway. It would have been just as valid to set the threshold at 0.04992 (bit of an odd choice) before performing the NHST, so the difference isn't really meaningful (except in what it tells us about our self-skepticism).

You could always just report the p-value and let the reader draw their own conclusions (i.e. not reject or accept anything).

$\endgroup$
-6
$\begingroup$

The answer is absolutely not. There is no "in the eye of the beholder", there is no argument, the answer is no, your data is not significant at the $p=0.05$ level. (Ok, there is one way out, but its a very narrow path.)

The key problem is this phrase: "We came across some data...".

This suggests that you looked at several other statistical hypothesis, and rejected them because they did not reach your significance level. You found one hypothesis that (barely) met your standard, and you are wondering whether it is significant. Unless your $p$ value accounts for such multiple hypothesis testing, it is overly optimistic. Given that you are just three decimal points away from your threshold, considering even one additional hypothesis would surely push $p$ over the line.

There is a name for this sort of statistical malfeasance: data dredging. I'm ambivalent about reporting it in the paper as an interesting hypothesis; does it have some physical reason you expect it to hold?

There is, however, one way out. Perhaps you decided a priori to perform just this one test on just this one data set. You wrote that down in your lab notebook, in front of someone so that you could prove it later. Then you did your test.

If you did this, then your result is valid at the $p=0.05$ level, and you can back it up to skeptics like me. Otherwise, sorry, it is not a statistically significant result.

$\endgroup$
9
  • 14
    $\begingroup$ This may be over-relying on a particular choice of phrasing; you're assuming rather a lot from what might be simply a poor choice of words - not everyone here has English as a first language. It's definitely worth raising as a potential problem, but to simply state things so baldly ("absolutely not") implies you know more than we can tell from what's here. (Further, the reference to a 'lab notebook' implies the OP is doing work in a lab. I doubt this is the case. Again, you imply you know more than we have here.) $\endgroup$
    – Glen_b
    Oct 12, 2013 at 20:56
  • 10
    $\begingroup$ Mike McCoy, thank you for your answer, but I'm afraid in this case Glen_b is correct. I am not a native English speaker, and while I strive to write and speak as fluently as my skills allow, usage and connotation continue to elude me. So, in this particular case, we didn't try different things until we found something that was significant. Actually, what we were trying to prove is that there were no statistically significant increase in some error value, and in one particular case we found that the error was actually reduced, and when we ran the W test, this is where we got the 0.0499. $\endgroup$ Oct 13, 2013 at 0:25
  • 1
    $\begingroup$ Mike, I also did not see a problem in the phrasing of the question. And it seems nobody else saw signs of data snooping, mining, dredging, whatsoever here ... And it definitely lies in the eye of the beholder. There is no mathematical fact but a decision rule chosen by the statistician. Re-read what AlefSin, Glen in his point (2) and I wrote. $\endgroup$ Oct 13, 2013 at 6:56
  • 3
    $\begingroup$ @IslamEl-Nabarawy If you wanted to establish equivalence/lack of difference, you have many other problems than how to interpret a value close to the threshold or potential data snooping. Just finding a p-value slightly over .05 (or whatever error level you choose) is definitely not enough. Look up “testing for equivalence” here and elsewhere or ask a question specifically about that because it's an entirely different problem. $\endgroup$
    – Gala
    Oct 13, 2013 at 10:32
  • 4
    $\begingroup$ "There is, however, one way out. Perhaps you decided a priori to perform just this one test on just this one data set. You wrote that down in your lab notebook, in front of someone so that you could prove it later. Then you did your test. If you did this, then your result is valid at the p=0.05 level, and you can back it up to skeptics like me. Otherwise, sorry, it is not a statistically significant result" Talk about guilty until proven innocent. So, in the absence of forensic evidence ruling out academic dishonesty, an analysis is worthless? Sheesh. $\endgroup$ Mar 3, 2017 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.