# Is p=5.0% significant?

Today I was asked, whether a p-value of 0.05 (exactly) is considered significant (given alpha=5%) or not. I did not know the answer and Google turned up both answers: (a) the result is significant if p is less than 5% and (b) if p is less than 5% or equal to 5%.

Of course, none of these websites cited anyone. Why should one - it's common knowledge and the 5% are arbitrary, anyway. But that won't help me to tell my students something to remember.

So, here are my desperate questions on testing hypotheses: If the p-value is exactly the alpha - do I consider the result significant or not? And what is the authorative citation in this case?

Thank you very much

-

Leaving aside some practical issues (such as the extent to which $\alpha$ is arbitrary, for example), the definitions of significance level and p-value make the answer to this question unambiguous.

Which is to say, formally, the rejection rule is that you reject when $p = \alpha$.

It really should only matter for the discrete case, but in that situation, if you don't reject when $p=\alpha$, your type I error rate won't actually be $\alpha$!

(As far as I am concerned there's no 'authoritative' citation; you really need to get to grips with both the Neyman-Pearson and the Fisherian approaches to hypothesis testing, and it's something that developed over time.)

There are any number of good statistics texts that describe hypothesis testing correctly.

The definition of p-value is given correctly in the first sentence of the relevant Wikipedia article*:

the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.

*(and no, wikipedia isn't an authority, I am just saying that the definition is right)

For simplicity, let's stick with point nulls; it serves to get the point across without muddying the waters with additional issues.

Now the significance level, $\alpha$ is the selected type I error rate. This is the rate which you choose the null hypothesis to be rejected when it's true. That is, it's the proportion of the time you should reject the null. Now consider a test statistic with a discrete distribution - the only time a $p$ of exactly $\alpha$ is actually possible**. (It will also typically be the case that the actual alpha will be different from something nice and round like 5%.)

** Well I guess I am confining my discussion to only purely-discrete- or purely-continuous- distributed test-statistics. In the mixed case, you can figure out how my discrete discussion applies (in the situations when it applies).

e.g. consider a two-tailed sign test with $n=17$, say. The nearest achievable significance level to 5% is 4.904%. So let's choose $\alpha = 4.904\%$ (or to be more precise, $\frac{137500}{2^{17}}$).

So when $H_0$ is true, what's the rejection rate if we reject when $p=\alpha$? We can work it out. It's 4.904% - it's the $\alpha$ we chose.

On the other hand, when $H_0$ is true, what's the rejection rate if we don't reject when $p=\alpha$? We can work it out. It's only 1.27%. It's way less than $\alpha$. That's not the test we signed up for!

That is, our tests (quite plainly!) have the desired properties if $p=\alpha$ is in the rejection region.

[Now let's consider your situation. Is your p-value actually exactly 5%? I bet it isn't exactly that, for several different reasons. But in any case, you can state that formally, $p=\alpha$ is a rejection.]

If you describe your rejection rule up front and show that (if the assumptions are satisfied), it has the desired significance level, then there's probably no need for references.

A rejection rule is simply a statement about which values of the test statistic will cause you to reject $H_0$. It's equivalent to defining the rejection region (for which see Casella and Berger, Statistical Inference, p346, which defines the term rejection region in plain terms).

The same book defines p-values (p364) in different terms to wikipedia (but the same resulting meaning) -- that is it defines it as (for a given data set), the smallest $\alpha$ that would lead to rejection of the null.

(If you have a different edition the page numbers may change, but it has an index, so you can look up terms; take care, you may need to look at the listings under 'Hypothesis testing' or something similar in the index to find 'rejection region')

Hmm, let's try another book off the shelf. Wackerly, Mendenhall & Scheaffer Mathematical Statistics with Applications, 5th edition, defines a rejection region on p412 and a p-value (same def as C&B) on p431.

-
Thank you. Please allow me the stupid question of a "practical user of statistics": Where except Wikipedia will I find this definition and/or rejection rule, you refer to? I screened some statistic books, but did not find the passage. Probably searching for the wrong words... –  BurninLeo Dec 16 '13 at 21:00