Do we reject the null hypothesis if the p-value is the exact same as our level of significance α?
For example, with α = 0.05, we observe p = 0.05.
Should we reject? Or, do we reject only if p is strictly smaller than α?
$\begingroup$
$\endgroup$
-
$\begingroup$ Note: if our random variable is discrete, we might have Pr(p = α) > 0. In other words, we could actually have a non-zero chance of observing p = α. $\endgroup$ – Guillaume F. Nov 25 '20 at 4:33
-
1$\begingroup$ Coverered on site a few times already. For a level $\alpha$ test you need the actual type I error rate $\leq\alpha$. The most powerful test using some given test statistic that satisfies that would be when you include the critical value in the rejection region. This corresponds to rejecting when $p=\alpha$. The extreme case is most illustrative - situations where you can get type I error rate of exactly $\alpha$ but the next smallest attainable significance level is $0$, which you're then stuck with if you don't reject when $p=\alpha$. $\endgroup$ – Glen_b Nov 25 '20 at 5:23
-
1$\begingroup$ See also: Do you reject the null hypothesis when p < α or p ≤ α? $\endgroup$ – Guillaume F. Dec 4 '20 at 6:39
$\begingroup$
$\endgroup$
The common thresholds of 0.1, 0.05, and 0.01 that p-values are evaluated against are intended to be heuristics rather than steadfast rules. The smaller the p-value the better, the less likely it is for the null hypothesis to be observed in the data. Therefore, these thresholds are not meant to represent strict "cutoffs" where decisions are based only on whether a p-value passes that specific cutoff. For more details on interpretation of p-values, please see Wasserstein & Lazar (2016).
Your best recourse would be to indicate that your statistical model has a pretty low p-value and although it doesn't entirely cross the threshold of 0.05, there is generally enough evidence to reject the null hypothesis.
$\begingroup$
$\endgroup$
A good answer from Anavir. In practice, the value of $\alpha$ one uses is quite arbitrary.
To address your issue more directly, though, the answer is it doesn't matter!
Why? For simplicity's sake, we're going to assume we're working with simple hypotheses, with continuous distributions specified under the null and alternative hypotheses. When we "fix $\alpha$" we really ensure that $Pr(\text{rejecting } H_0 | H_0 \text{ is true}) \leq \alpha$.
For continuous real-valued random variable $X$ and $x \in \mathbb{R}$, as I'm sure you know, $Pr(X = x) = 0$. Also, notice that the $p$-value, which we'll denote as $P$ is a continuous random variable in and of itself! (In fact, under the null in this case, its a uniform random variable on $[0,1]$, but that's besides the point). The $p$-value which we observed, which we will denote as $p$ is a realization of $P$.
If $Pr(P \leq \alpha) = \alpha$, then $$Pr(P \leq \alpha) = Pr(P = p) + Pr(P < \alpha) = Pr(P < \alpha) = \alpha$$.
Indeed, rejecting when your p-value is less than or equal to $\alpha$, or strictly less than $\alpha$, makes no difference. We still satisfy the constraints we set out for ourselves.
-
$\begingroup$ I disagree that it does not matter; for a discrete random variable, such as a binomial, we can have a non-zero probability that p = α exactly. $\endgroup$ – Guillaume F. Nov 25 '20 at 4:26
-
$\begingroup$ Sure. I assumed a continuous case for simplicity in my original answer, so what I say holds. When one has a discrete random variable (or even hybrid random variable) like you say, then this concern is valid. Then one might resort to more "sophisticated considerations". $\endgroup$ – user303375 Nov 25 '20 at 4:47
-
$\begingroup$ In that case, when your p-value is exactly equal to $\alpha$ you do reject. You should also check out randomized (not randomization) tests, which can be used when we don't "use up" all of our $\alpha$. $\endgroup$ – user303375 Nov 25 '20 at 4:58
