# Is p=5.0% significant?

Today I was asked, whether a p-value of 0.05 (exactly) is considered significant (given alpha=5%) or not. I did not know the answer and Google turned up both answers: (a) the result is significant if p is less than 5% and (b) if p is less than 5% or equal to 5%.

Of course, none of these websites cited anyone. Why should one - it's common knowledge and the 5% are arbitrary, anyway. But that won't help me to tell my students something to remember.

So, here are my desperate questions on testing hypotheses: If the p-value is exactly the alpha - do I consider the result significant or not? And what is the authorative citation in this case?

Thank you very much

Leaving aside some practical issues (such as the extent to which $\alpha$ is arbitrary, for example), the definitions of significance level and p-value make the answer to this question unambiguous.

Which is to say, formally, the rejection rule is that you reject when $p = \alpha$.

It really should only matter for the discrete case, but in that situation, if you don't reject when $p=\alpha$, your type I error rate won't actually be $\alpha$!

(As far as I am concerned there's no 'authoritative' citation; you really need to get to grips with both the Neyman-Pearson and the Fisherian approaches to hypothesis testing, and it's something that developed over time.)

There are any number of good statistics texts that describe hypothesis testing correctly.

The definition of p-value is given correctly in the first sentence of the relevant Wikipedia article*:

the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.

*(and no, wikipedia isn't an authority, I am just saying that the definition is right)

For simplicity, let's stick with point nulls; it serves to get the point across without muddying the waters with additional issues.

Now the significance level, $\alpha$ is the selected type I error rate. This is the rate which you choose the null hypothesis to be rejected when it's true. That is, it's the proportion of the time you should reject the null. Now consider a test statistic with a discrete distribution - the only time a $p$ of exactly $\alpha$ is actually possible**. (It will also typically be the case that the actual alpha will be different from something nice and round like 5%.)

** Well I guess I am confining my discussion to only purely-discrete- or purely-continuous- distributed test-statistics. In the mixed case, you can figure out how my discrete discussion applies (in the situations when it applies).

e.g. consider a two-tailed sign test with $n=17$, say. The nearest achievable significance level to 5% is 4.904%. So let's choose $\alpha = 4.904\%$ (or to be more precise, $\frac{137500}{2^{17}}$).

So when $H_0$ is true, what's the rejection rate if we reject when $p=\alpha$? We can work it out. It's 4.904% - it's the $\alpha$ we chose.

On the other hand, when $H_0$ is true, what's the rejection rate if we don't reject when $p=\alpha$? We can work it out. It's only 1.27%. It's way less than $\alpha$. That's not the test we signed up for!

That is, our tests (quite plainly!) have the desired properties if $p=\alpha$ is in the rejection region.

[Now let's consider your situation. Is your p-value actually exactly 5%? I bet it isn't exactly that, for several different reasons. But in any case, you can state that formally, $p=\alpha$ is a rejection.]

If you describe your rejection rule up front and show that (if the assumptions are satisfied), it has the desired significance level, then there's probably no need for references.

A rejection rule is simply a statement about which values of the test statistic will cause you to reject $H_0$. It's equivalent to defining the rejection region (for which see Casella and Berger, Statistical Inference, p346, which defines the term rejection region in plain terms).

The same book defines p-values (p364) in different terms to wikipedia (but the same resulting meaning) -- that is it defines it as (for a given data set), the smallest $\alpha$ that would lead to rejection of the null.

(If you have a different edition the page numbers may change, but it has an index, so you can look up terms; take care, you may need to look at the listings under 'Hypothesis testing' or something similar in the index to find 'rejection region')

Hmm, let's try another book off the shelf. Wackerly, Mendenhall & Scheaffer Mathematical Statistics with Applications, 5th edition, defines a rejection region on p412 and a p-value (same def as C&B) on p431.

• Thank you. Please allow me the stupid question of a "practical user of statistics": Where except Wikipedia will I find this definition and/or rejection rule, you refer to? I screened some statistic books, but did not find the passage. Probably searching for the wrong words... – BurninLeo Dec 16 '13 at 21:00
• see updates to answer – Glen_b Dec 16 '13 at 22:58
• You're great! Sure, you knew that, but sometimes it's worth to state anyway. Thanks a lot! – BurninLeo Dec 17 '13 at 18:39
• (+1) Really nice answer. – chl Dec 18 '13 at 17:01

An interesting confession I had learned in my early biostatistics class from a professor is that the 0.05 significance level was more so arrived through a consensus rather than golden truth. Since then, I have seen literature that flirts with the 0.05 significance level, such as "approaching" to still be a striking finding of the study and I have heard arguments that the 0.05 significance level may not apply to all fields of research. With that said, I have found point estimates and confidence intervals to be more informative than significance levels. Here's an interesting article on the matter (to me anyway).

• Thanks for the reference @user2300643. And thanks for avoiding the financial barrier to access! – doug.numbers Dec 16 '13 at 23:00
• There's an interesting discussion of the origins of the 5% etc significance levels here. – Glen_b Dec 17 '13 at 1:57

The value of p is usually set for consensus as said previously ( or rather laziness). To really be able to say that something is significant we must find the value of p that corresponds to the effect size, sample size, and how stringent you want it to be for your data. This is called power analysis (it's a subfield within statistics). Many people are either not aware or simply don't use it because it's not straightforward. This is not to say it's fine the way it is. We should always do this kind of study to draw inferences that are really meaningful.