Do we reject null if we have a pvalue of 0.0503 at alpha = 0.05? What way to validate the rejection, if ever?

Question

I know it is more than 0.05, but I'm just wondering since rounding it to two decimals will give 0.05. I just wanted to ensure I am not falsely accepting the null hypothesis. Is there any way to say there is enough evidence to reject null even with this value close to the alpha? Also, I know I'm not supposed to, but upon doing post hoc, I got a result of means among treatments that were different.

Answer 1

It sounds like the only reason you care about the overall p-value from ANOVA is to justify followup tests to look at differences between various groups (accounting for multiple comparisons). BUT those tests are valid even if the overall p-value is >0.05. Below is a link is a prior question that asks this question and provides several answers. So my advice is: Ignore the overall p-value and run sensible followup multiple comparison tests.

Is it ok to run post hoc comparisons if ANOVA is nearly significant?

Answer 2

If you're looking for a strict reject /not reject decision (as people typically would be for say a research publication where in many areas the Neyman-Pearson approach is common convention), then you follow the rule on which that decision is based (and conventionally you don't reject the null).

You'd normally reject if $p$ was exactly $\alpha$ (maintaining a type I error rate not exceeding $\alpha$), but not if it merely rounds to $\alpha$. You should only see exact-$α$ outcomes with discrete test statistics (and even then, pretty rarely unless sample sizes are very small). [You can't claim $α$ is an upper bound on the probability of a type I error in a particular circumstance if you reject more cases than your rule says to in order to attain the bound.]

Yes, hard yes/no decisions are indeed arbitrary near the threshold, by their nature. That's the case for almost any binary decision based on a continuous quantity, it's not unique to hypothesis tests.

If you want to claim that your $α$ levels (and hence, $p$ values) mean what people define them to mean, this is how it works -- if you're not rejecting nulls at all, then you simply don't have a type I error rate; if you do reject some nulls, then you need to be clear what your rejection rule is and hence what $α$ that implies.

If instead you decide beforehand that don't want a hard threshold, and you want others to come along with you, you would need to be clear about what you're replacing it with and what its properties would be.

If you adopt the policy that you can reject if it rounds to $0.050$ and you should only fail to reject if it rounds to $0.051$ say, then all you've done is shifted the rejection region from $p\leq 0.05$ to $p< 0.0505$. Your significance level is not what you started with, but larger.

Which is fine if you're clear about what it actually is.

But then if you choose that new decision rule, that new cut off is just as arbitrary, so soon enough someone would say "why should I not then reject if it's just above that new threshold?" It's not a different problem in that new scheme, you then have the same arbitrariness 'issue' in a slightly different place.

It's then just moving goalposts, and your actual significance level is just going to land wherever you finally decide to stop. Beware the charade of having a firm decision when it comes out how you wanted and "not making a firm decision at an arbitrary threshold" otherwise. If you're really in the business of making a decision to reject or not, actually pick a rejection rule you will keep to - and whose actual properties can (finally!) be evaluated.

Of course if you are in some other circumstance, and aren't seeking a hard binary decision, you needn't use such a decision rule. However, if you do use some other strategy for evaluating hypotheses, I'd encourage you to consider the properties of the entire approach you use for whatever it is you're using it to do. You're free to make your decisions however you like as long as you don't give the impression that they're something they're not.

For one example, if you wanted to have not two but three possibilities ("reject $H_0$", "fail to reject $H_0$", "continue sampling"), then you might want to look into sequential analysis as one possibility of arranging things, where the consequences of that scheme have been investigated. That's fine if that's what you seek to do.

Or if you wanted your study to form part of a continuing collection of evidence, perhaps in a Bayesian context, that's fine too.

Whatever you choose to do, you would need to define how your protocol is going to work before you see data. That is, if you're making a strict decision, write your decision rule down. If you're not making a strict decision, write that down too, so you're then not tempted to decide to actually state that you do reject after all, because $p$ came out to $0.038$ or something. You can't be in a sort of Schrödinger's cat setting where a rejection rule is in a superposition of both being in place and not until you see the data.

If your process is not explicit in advance, it's quite easy to end up, albeit unintentionally, just engaging a more involved form of p hacking. There's many ways for this to happen.

Answer 3

1. This is a good example of the problems of using strict thresholds for inference Murtaugh (2014), "In defense of P values", Ecology, 95, 3.

There is nothing sacred about 0.05. p-values are on a continuous scale and the interpretations of them also should be continuous rather than a binary "significant/non-significant".

The above paper reprints a useful guide from Ramsey and Schafer (2002), The Statistical Sleuth: A Course in Methods of Data Analysis to give some idea on how you should interpret p-values in this fashion.

In this context, it might also be important to interpret your results based on effect sizes. Does your effect size mean biologically meaningful differences? If you have a huge effect size and an almost significant p-value, I would be even more confident in interpreting it as a significant result.

TL;DR: I would interpret this as significant, but convincing your reviewers might be a different question altogether.

2. I think you might have to give more info on the kind of study you are doing. But to my understanding, post hoc tests generally are more conservative, and if you got a p-value very close to but above 0.05, chances are you won't see significance in your post hoc test.

Answer 4

The idea of using a threshold to "reject" comes from the Neyman-Pearson school of thought. Following such an approach to statistics allows us to make some interesting claims, particularly regarding sample size, power, and effect size. Play with the pwr package in R to see how these interact with the threshold, called alpha in that package.

However, such an approach turns the problem into a binary decision: either the p-value is above the threshold and we do not reject, or we reject. In order to get the desirable properties from a Neyman-Pearson approach, we have to be rigid about this decision rule. Otherwise, we can wind up in the position where I have found myself where practitioners (customers of mine) want to reject when $p=0.0503$ because "it's so close" but do not want to reject when $p = 0.0497$ because "it's so close". Yes, they wind up seeing a higher p-value as stronger evidence against the null. Weird...

The decision rule that gives the desirable properties we get from Neyman-Pearson frequentist statistics is to have a binary decision based on a threshold: above the threshold or not. Once you deviate from the Neyman-Pearson approach, you no longer have the desirable properties it gives.

Therefore, if you care to compare to a threshold in this Neyman-Pearson approach, I would say to be rigid about it. Whether or not you should be comparing to a threshold at all is a separate matter.

Answer 5

The "accept/reject" terminology is connected to acceptance sampling in product quality control, and it may well be that this is what Neyman and Pearson had originally in mind when coming up with this terminology. "Accept" then means that the batch of product from which the sample is taken is deemed OK for being sold. This is a practical decision, which is in fact binary - if you "accept", you do one thing, if you "reject", you do another.

Note particularly that "accept" does not mean that you should believe that the null hypothesis is exactly true. It just means (in the quality control setup) that your quality check didn't indicate against it, which is actually (about) as good as it gets without measuring all individuals, which may be expensive or even destructive. It is a basis for a decision about whether you should ship your product, and if then in fact in turns out that there is indeed a somewhat too large percentage of faults in the products in case you "accept" (type II error, the probability for which can get very close to 95% if deviation from the null hypothesis is small), you can at least say that you followed a reasonable protocol for your decision, and chances are there are not many more faults than there should be. Neyman, one of the originators of classical statistical tests, had a philosophy that he called "inductive behaviour", making it quite clear that he thought of accept/reject decisions in terms of behaviour rather than believing what is true or not.

It is misleading to apply this terminology and particularly "accept" to the truth of scientific claims that are not connected to binary decisions regarding practical actions. Not rejecting a null hypothesis never means that you should believe the null hypothesis to be true, particularly because the null hypothesis is an idealised formal model of a reality that is much more complicated than that anyway.

A p-value of 0.0503 indeed doesn't say much different from a p-value of 0.049, and a more appropriate interpretation is gradual, something like "there is somewhat weak and inconclusive evidence against the null hypothesis" as already said in other answers. However, if you want to use the term "significant", you have to play by its rules, which means fixing a rejection threshold before seeing the data. The advantage of the otherwise rather arbitrary $\alpha=0.05$ is that even if you haven't explicitly decided on a level before seeing the data, people will be fine with using 0.05 because that's what pretty much everybody does, and therefore people will buy that you did not choose this in a data dependent manner (even though in fact this may not be true if based on the data you would have been prepared to choose something else). But then if you end up with $p>0.05$ the result is not significant and you can't change that by saying something like "0.0503 is 0.05 rounded". This is the consequence of buying into the binary "significance/accept/reject" terminology. You may want to avoid doing that (not only here, but generally).

You may wonder whether this is even of any interest if we know that the null hypothesis isn't literally true anyway, but in fact I'd say it may well be, because it is about whether the data clearly are incompatible with an idealised thought construct that has a certain straightforward scientific interpretation, which is informative, particularly because the effect size/test statistic will say in what way the data indicate against it (for example "the mean is larger than it should be").

The most important thing to understand here is that it is not your job to "decide" whether the null hypothesis looks true or not in a binary manner. The message in the data is what it is, it doesn't rely on your decision. If it is inconclusive, so be it. You may in some situations have to decide about a binary course of action, but that is a different matter, see above.

Note also that the binary result of a significance test (better called "reject" vs. "not reject") is about the relation between the null hypothesis and your specific data set rather than about the truth of the null hypothesis (which is not true anyway as "all models are wrong (but some are useful)". This means that you can't really "validate" it on other data, although of course analysing more data may in fact help you find out more about your research hypothesis of interest and may eventually give you more confidence saying that the null hypothesis is surely no longer compatible with all the evidence you have (although bringing together results from different experiments is not exactly a piece of cake, particularly not in "classical" frequentist statistics using the standard tests and confidence intervals).

Answer 6

Strictly speaking, no. This is a very good question because it strikes at the heart of frequentist hypothesis testing and what it actually means. As the investigator, you decide on alpha before conducting the experiment. This is an arbitrary decision. It could be guided by some standards, such as your research field or industry, but fundamentally it is a decision rule you set for yourself that answers the question "How much should the statistic I get be extreme enough for me to reject the notion that my data comes from my null hypothesis assumption?" Once you have your data, then you cannot change your alpha anymore. Under the rules that you set for yourself for the experiment, you cannot reject your null hypothesis.

HOWEVER, does this mean that the evidence you found against the null are not valid? NO! Of course not. You can see with your own eyes that there is evidence for the alternative. A statistical test does not invalidate what your eyes are seeing. You just cannot claim that you found this statistically significant (which is shorthand for, I decided on alpha, I did the experiment, I computed the p-value, I compared it against my preset alpha, and I saw that it is less than alpha). Saying so would be lying, and substituting things like "approaching significance," "almost significant," etc. also don't make sense because the hypothesis test is a yes or no question, there is no "almost significant."

What can you do? Report everything. Report your alpha, report your p-value, report that you were not able to reject your null. Report the size of the effect you found. Report all of this and let your readers decide. Be transparent so that people who read your work will know exactly what happened and how to consider your results in the context of what they might be thinking about or doing themselves.

In the specific context of doing post-hoc tests after ANOVA, the same rules apply. When you decided to do ANOVA, you were under the prevailing assumption that you do not know where the signals are, but suspect that they can be found using the multiple possible post-hoc tests which only come into play if the global ANOVA is rejected. If you fail to reject the global test, then you cannot do the post-hoc tests. You should note that in this design, the post-hoc test relies on rejecting the global test in order for them to account for multiple comparisons. That is, if you fail to reject the null but do Tukey's test anyway, any rejections of post-hoc nulls using Tukey wrongly benefits from falsely assuming that the global null was not rejected. Now, you can instead do all your initially proposed post-hoc tests and adjust for multiple comparisons directly instead of doing ANOVA altogether. But here, you must still report both results as well as report that you resorted to doing multiple comparisons after failing to reject the global null from ANOVA.

Long story short: Be honest. That is a critical component to hypothesis tests working the way they should be.

Answer 7

I just wanted to ensure I am not falsely accepting the null hypothesis. Is there any way to say there is enough evidence to reject null even with this value close to the alpha?

• to ensure

  There is no certainty or guarantee. You will never know certainly whether ot not the null hypothesis is true or not.

  The reason for hypothesis tests is to make practical dichotomic decisions. If x>z do A, if x≤z do B.

• accepting the null hypothesis

  We normallly don't accept the null hypothesis. Instead we do not reject it. It is a small nuance.

  Why can't we accept the null hypothesis, but we can accept the alternative hypothesis?

  In terms of two one-sided t-tests (TOST), one way to 'accept' the null hypothesis is to reject the hypothesis that the difference is larger than a certain amount. But it is not exactly the same as accepting the null hypothesis.

• The alpha level of 0.05 is arbitrary. There is no solid foundation why you should use this. It is a rule of thumb.

  What is the history of $p < 0.05$ or 95% confidence?

• Still, if you do settle on a certain alpha/significance level, then yes, it is a hard boundary. One side of the boundary is accepted, the other side of the boundary is rejection.

  This is a practice in every day life where yes/no decisions need to be made.

  This is not always the case. Decisions can be alsonof the type yes/no/undecided. But still, there will be hard boundaries.

Answer 8

Lots of good answers already - I'm not adding anything new here just presenting it slightly differently. I would approach the question by asking the OP:

Why did you calculate a p-value in the first place?

The answer should automatically resolve the original question and I can think of the following possibilities:

Option 1: You need to make a binary decision about, for example, a quality control (QC) procedure where p >= 0.05 indicates QC passed and p < 0.05 QC failed. For this you have done some power analysis before data collection to decide on sample size, effect size, type I error (false positive) and II error (false negative). Using the traditional critical value of 0.05 for significance your result of 0.0503 is "not significant", QC passed.

Option 2: You calculated a p-value to get a sense for how much evidence the data provides against the null hypothesis. In this case, a p-value is a summary statistics and there is no need to stamp it as significant or not. In fact, that would make the presentation of results more confusing than helpful. To pre-empt a reviewer's criticism that p = 0.0503 is "not significant" you could/should state upfront that, yes, 0.0503 provides only weak evidence against the null, but in addition to other reasons results are still interesting (if that is the case of course). It seems to me that many p-values reported in scientific pubblications (in biology at least) fall in this category even if authors categorize them as significant or not.

Option 3: This is somewhat an hybrid of the previous two options. You haven't done any formal power analysis ahead of the experiment but you still need to make a binary decision on what to do next. This is often the case in research where you decide what to do next after seeing the results from the current experiment. Like option 2, there is no need to stamp results as significant or not. p = 0.0503 may warrant further investigation, but you would do so being aware that you may be chasing noise. Also, keep in mind that filtering results based on p-value will cause over-estimation of effect size. The more stringent the p-value cutoff is the more exaggerated is the estimate of effect size.