In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was true. The exact probability of observing a sample score or more extreme under the null is the $p$ value. More generally we reject if $\alpha>p$.

I am now wondering about the following. The $p$ value seems to give an exact estimate of the probability of falsely rejecting a true null hypothesis (if we decide to do so), which is akin to the Type I error definition. Since we know (estimate) the probability of observing the sample score (or more extreme values), $\alpha$ seems to be a maximum acceptable Type I error, whereas $p$ is exact. Put differently it appears to give the minimum $\alpha$ level under which we could still reject the null.

Is this correct?


[Assume, for the moment that we're not talking about composite null hypotheses, since it will simplify the discussion to stick to the simpler case. Similar points could be made in the composite case but the resulting additional discussion would be likely to prove less illuminating]

The probability of a type I error, which (if the assumptions hold) is given by $\alpha$ is probability under the notion of repeated sampling. If you collect data many times when the null is true, in the long run a proportion of $\alpha$ of those times you would reject. In effect it tells you the probability of a Type I error before you sample.

The p-value is instance-specific and conditional. It does not tell you the probability of a type I error, either before you sample (it can't tell you that, since it depends on the sample), or after:

If $p\geq\alpha$ then the chance you made a Type I error is zero.

If the null is true and $p<\alpha$ then the chance you made a Type I error is 1.

Take another look at the two things under discussion:

  • P(Type I error) = P(reject H$_0$|H$_0$ true)

  • p-value = P(sample result at least as extreme as the observed sample value|H$_0$ true, sample)

They're distinct things.

Edit - It appears from comments that it is necessary to address your second paragraph in detail:

The p value seems to give an exact estimate of the probability of falsely rejecting a true null hypothesis

Not so, as discussed above. (I assumed this was sufficient to render the rest of the question moot.)

α seems to be a maximum acceptable Type I error,

In effect, yes (though of course we may choose a lower $\alpha$ that the absolute maximum rate we'd be prepared to accept for a variety of reasons).

whereas p is exact.

Again, not so; it's not equivalent to $\alpha$ in the suggested sense. As I suggest, both the numerator and denominator in the conditional probability differ from the ones for $\alpha$.

Put differently it appears to give the minimum α level under which we could still reject the null.

In spite of my earlier caveats, there is a direct (and not necessarily particularly interesting) sense in which this is true. Note that $\alpha$ is chosen before the test, $p$ is observed after, so it's necessary to shift from our usual situation.

If we posit the following counterfactual:

  • we have a collection of hypothesis testers, each operating at their own significance level

  • they are each presented with the same set of data

then it is the case that the p-value is a dividing line between those testers that reject and those that accept. In that sense, the p-value is the minimum α level under which testers could still reject the null. But in a real testing situation, $\alpha$ is fixed, not variable, and the probability we're dealing with is either 0 or 1 (in a somewhat similar sense to the way people say "the probability that the confidence interval includes the parameter").

Our probability statements refer to repeated sampling; if we posit a collection of testers each with their individual $\alpha$, and consider only a single data set to test one, it's not clear $\alpha$ is the probability of anything in that scenario - rather, $\alpha$ represents something if we had a collection of testers and repeated sampling where the null is true - they'd each be rejecting a proportion $\alpha$ of their nulls across samples, while $p$ would represent something about each sample.

  • 1
    $\begingroup$ Thank you for an interesting and thought-provoking post. It doesn't seem to get all the way to an answer, though. The ultimate question is whether "[the $p$ value] gives the minimum $\alpha$ level under which we could still reject the null." There's an implicit hypothetical there: if our $\alpha$ were less than the observed $p$-value, then we would not reject the null; but if our $\alpha$ were greater than the observed $p$-value, then we would reject the null. Are you suggesting this is not correct? $\endgroup$ – whuber Dec 19 '14 at 0:07
  • $\begingroup$ Didn't Glen implicitly answered that there is no such relationship? Therefore, there wouldn't be anything like minimum level as stated. $\endgroup$ – SmallChess Dec 19 '14 at 3:18
  • $\begingroup$ @whuber I didn't address the specifics of that because as I read it, the last paragraph is predicated on an assumption ("the p-value seems to ...") that my answer explains to be false, which renders the rest of the paragraph moot without a rewording. If we leave that assumption aside, I take Alexis' answer to have covered that part, but I could address it. $\endgroup$ – Glen_b Dec 19 '14 at 3:53
  • $\begingroup$ @StudentT I didn't read my answer as implying that. I have edited my answer. $\endgroup$ – Glen_b Dec 19 '14 at 4:20
  • $\begingroup$ I think I like the hypothetical collection of testers rather than my imagining tests with various $\alpha$ levels. $\endgroup$ – John Dec 19 '14 at 6:01

Your interpretation seems about correct. The caveat I would add is that $\alpha$ is an a priori decision to be made before conducting an hypothesis test. So it's no good finding that the p-value for a test statistic is, say, 0.00021, and then reporting that your test had an $\alpha$ of 0.00021; that would make $\alpha$ and p (falsely) synonymous.

  • $\begingroup$ Are you suggesting they could say $\alpha = 0.001$ if $p = 0.00021$ ? $\endgroup$ – John Dec 18 '14 at 21:27
  • $\begingroup$ Thanks - but apparent contradiction to @John post above. Mind to elaborate? $\endgroup$ – tomka Dec 18 '14 at 21:29
  • $\begingroup$ @John You are losing the temporality in my answer: $\alpha$ is a priori it is a decision made before the test and cannot be dependent on p without changing the meaning (as opposed to value) of $\alpha$. By contrast the p-value must be a consequence of the test and data. If you say that you can just select $\alpha$ base d on p, how is that different that saying "I reject all tests at the p-level?" (Maybe not very useful!) $\endgroup$ – Alexis Dec 18 '14 at 22:55
  • $\begingroup$ I'm just trying to clarify. Perhaps I should have asked... what about the questioner's interpretation seems correct given that the questioner was interpreting $\alpha = p$ both as marginal criterion and Type I error? $\endgroup$ – John Dec 19 '14 at 6:13
  • $\begingroup$ @John Oh! I see. Most of the understanding up until the last few sentences. $\endgroup$ – Alexis Dec 19 '14 at 9:44

The $p$-value is not "an exact estimate of the probability of falsely rejecting a true null hypothesis". This probability is fixed by construction of an $\alpha$-level test. Rather it is an estimate of the probability that other realisations of the experiment are more extreme than the actual realisation. Only if the present realisation belongs to the top $\alpha$ extreme realisations, we reject the null hypothesis.

But it is right that you can imagine the $p$-value to be the minimum $\alpha$, such that , if this $\alpha$ had been chosen this way, the test would be on the border of significance to insignificance for the present data.

Maybe a different explanation helps: We say that we reject the null hypothesis, iff the present outcomes can be shown to belong to the extreme $100 \alpha \%$ of possible outcomes, provided the null hypothesis holds. The $p$-value just indicates how extreme our outcomes actually are.

  • $\begingroup$ So if I imagine that (minimum $\alpha=p$), wouldn't $p$ then always indicate the probability of falsely rejecting the null? $\endgroup$ – tomka Dec 18 '14 at 18:33
  • 1
    $\begingroup$ I did not rate down - would be interested why the reader did so. $\endgroup$ – tomka Dec 18 '14 at 18:46
  • $\begingroup$ How could the $p$ be the minimum $\alpha$? What would the maximum be? Maybe you mean that it could have been the minimum criterion if you had selected it a priori. You can't change $\alpha$ after the finding has been observed. $\endgroup$ – John Dec 18 '14 at 19:53
  • $\begingroup$ I didn't say to change $\alpha$. The second paragraph is just the converse of the decision rule "reject iff $p<\alpha$". $\endgroup$ – Horst Grünbusch Dec 18 '14 at 21:53

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to a conditional probability connected to a single event (the data collected). In this case the latter is the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$. And, the $p$ is not the probability of falsely rejecting the null.

Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.

  • $\begingroup$ In cases involving continuous distributions of the test statistic, the "probability of observing the current data" is exactly zero. There also is an important subtlety with composite null hypotheses: $p$ is the supremum (taken over all elements of the null hypothesis) of the chance of observing data that lie in the critical region. $\endgroup$ – whuber Dec 18 '14 at 19:14
  • 1
    $\begingroup$ Sorry, slow editing, I'll probably still change it even more. But the comment is good to leave because it further clarifies what $p$ is. $\endgroup$ – John Dec 18 '14 at 19:26
  • $\begingroup$ I was confident you would continue to fix things up. :-) Thanks for continuing to work on this. Additional clarification of the third paragraph would be welcome--I am still puzzling over what you intend in saying "your decision to reject is independent of the $p$ value," since ordinarily the decision explicitly is based on comparing $p$ to $\alpha$. $\endgroup$ – whuber Dec 18 '14 at 19:29
  • $\begingroup$ I can fully follow the argument that it is the probability of observing the score or more extreme ones if the null is true. However, isn't it true that we often think of it in a way that sets $\alpha=p$ after the data have been observed, to see what the minimum $\alpha$ for rejecting the null would be, and at that moment the two become equivalent? $\endgroup$ – tomka Dec 18 '14 at 19:39
  • 1
    $\begingroup$ I think my thinking was strongly influenced by this idea. If I just consider alpha fixed prior to the experiment it is obvious that the two are only related via the decision rule $\alpha>p$. $\endgroup$ – tomka Dec 18 '14 at 19:45

When one calculates a $p$-value, one is actually computing a conditional probability in which the condition being assumed to be true is the null hypothesis. So in this way, the $p$-value is in a sense a quantifier of how likely we would expect to observe a sample at least as extreme as the one we saw, assuming the sample satisfies the distributional assumptions of the null hypothesis. This latter part is extremely important, because only then can we infer from the $p$-value whether or not there exists sufficient evidence to reject the null hypothesis.

If I give you a coin but tell you nothing about whether it is fair, and you toss it $100$ times and get $99$ heads and $1$ tail, you would very likely and reasonably conclude that the coin is not in fact fair. The way you would quantify this impression, as a statistician, is to first suppose that the coin is fair, and then demonstrate through the use of a binomial proportion test, that the chance that you could have gotten such an extreme result under that supposition is incredibly small--this value is the $p$-value of the test.

That said, even though that chance is astronomically small, it still isn't zero. There is a tiny, tiny possibility that a truly fair coin, tossed $100$ times, could give you $99$ heads and $1$ tail; or $99$ tails and one head; or all heads; or all tails, all due to random chance. Just because an event is rare does not mean it is impossible. Therefore, whenever we conduct a nontrivial statistical test, there is always some possibility of error. You would be quite confident that the coin isn't fair, but you could be wrong, and the probability you could be wrong in this sense is the Type I error.

  • $\begingroup$ That's right but I don't see how it answers the question. $\endgroup$ – John Dec 19 '14 at 6:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.