This question already has an answer here:

I am trying to understand the relationship of a p-value vs Type 1 error rate. My understanding of one interpretation of a p-value is the following: "the p-value tells us the probability of making a type 1 error, conditional on the fact that the null hypothesis is true and we do indeed decide to reject the null hypothesis".

Is this a correct interpretation or not? I am getting confused because I am reading from some sources which are claiming that p-value is NOT the same thing as a type 1 error. For example, see the following sources:

  • From this source there is the following quote: P = 0.05 does not mean there is a 5% chance of a Type I error (i.e. false positive).

There is also "A Dirty Dozen: Twelve P-Value Misconceptions" (Goodman 2008) which claims something similar.

Can anyone clarify my confusion?

Thank you!


marked as duplicate by gung, Silverfish, Peter Flom Jun 23 '15 at 22:10

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • You haven't provided enough information even to know what the "chance of a Type I error" would be: that depends on the level (usually termed $\alpha$) you have selected before even seeing the data. It seems that you are assuming some unstated connection between $\alpha$ and $p$. Could you explain what that is? Also, anticipating that a large number of answers might arise here attempting to explain p-values, and that they might be equally confusing, please first look at our thread on p-values. – whuber Jun 23 '15 at 20:22

A paper that may help is

Murdock, D, Tsai, Y, and Adcock, J (2008) P-Values are Random Variables. The American Statistician. (62) 242-245.

Imagine that you have a coin that you want to test if it is fair (maybe it is bent or otherwise distorted) and plan to flip the coin 10 times as your test. Clearly if you see 5 heads and 5 tails then you can't reject the null that it is fair and most people would be highly suspicious of the coin if you saw 10 heads (or 10 tails), but to be fair we should set up before the test a rule or rejection region to determine if we should reject the null hypothesis (fair coin) or not.

One approach to deciding on the rejection region is to set a limit on the type I error rate and choose the rejection region such that the most extreem values whose cumulative probabilities are less than the limit will constitute the rejection region. So if we use the traditional 0.05 as our cut-off then we can start with the extreems and see that if the coin is fair (null is true) then the probability of seeing 0, 1, 9, or 10 heads is less than 5%, but if we add in 2 or 8 heads then the combined probability goes above 5%, so we will reject the null if we see 0, 1, 9, or 10 heads and fail to reject otherwise.

A side note is that we could create a rejection region of reject if see 8 heads, don't reject otherwise and that would keep the probability of rejecting when the null is true under 5%, but it seems kind of silly to say we will reject fairness if we see 8 heads, but will not reject fairness if we see 9 or 10 heads. This is why the usual definitions of p-value include a phrase like "or more extreem".

So for our test we have our alpha ($\alpha$) level set at 5%, but the actual probability of a type I error (null true as part of the definition) is a little above 2% (the probability of a fair coin showing 0, 1, 9, or 10 heads in 10 flips).

Instead of comparing the actual number of heads to our rejection region, we can instead calculate the probability of what we observe (or more extreem) given the null is true and compare that probability to $\alpha = 0.05$. That probability is the p-value. So 0 or 10 heads would result in a p-value of $\frac{2}{1024}$ (one for 0, one for 10). 1 or 9 heads would give a p-value of $\frac{22}{1024}$ (one way to see 0, one way to see 10, 10 ways to see 1 and 10 ways to see 9). If we see 2 or 8 heads then the p-value is greater than 10%.

So to summarize: The probability of a Type I error is a property of the chosen cut-off $\alpha$ and the nature of the test (in cases like the t-test when all the assumptions hold, the probability of a type I error will be exactly equal to $\alpha$). The p-value is a random variable computed from the actual observed data that can be compared to $\alpha$ as one way of performing the test).

$\alpha$ is the threshold at which you determine that you will decide to reject your null hypothesis, if your pvalue is below that established $\alpha$. It does not mean you can accept the null hypothesis if you hit that value or below.

  • This is incorrect. The threshold beyond which you would reject your null hypothesis is $\alpha$, not the $p$-value. In addition, we never "accept the null hypothesis", & we would reject the null if $p<\alpha$. – gung Jun 23 '15 at 20:43
  • I stand corrected. I should have mentioned that the threshold I was talking about is the alpha and the p-value is what you would look to see lower than that value, as you mention. I should have re-read my comment. I'll make edits. – rvidal Jun 25 '15 at 15:28
  • After reading this edited answer, I went back through the question--twice--to see where it mentioned "accepting" the null hypothesis, but I couldn't find it. What aspect of this question are you trying to respond to, then? – whuber Jun 25 '15 at 16:28
  • I suppose I read too much into the question. Would you like I delete my entry? – rvidal Jun 25 '15 at 20:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.