What is the relationship between $p$ values and Type I errors [duplicate]

Question

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was true. The exact probability of observing a sample score or more extreme under the null is the $p$ value. More generally we reject if $\alpha>p$.

I am now wondering about the following. The $p$ value seems to give an exact estimate of the probability of falsely rejecting a true null hypothesis (if we decide to do so), which is akin to the Type I error definition. Since we know (estimate) the probability of observing the sample score (or more extreme values), $\alpha$ seems to be a maximum acceptable Type I error, whereas $p$ is exact. Put differently it appears to give the minimum $\alpha$ level under which we could still reject the null.

Is this correct?

Answer 1

[Assume, for the moment that we're not talking about composite null hypotheses, since it will simplify the discussion to stick to the simpler case. Similar points could be made in the composite case but the resulting additional discussion would be likely to prove less illuminating]

The probability of a type I error, which (if the assumptions hold) is given by $\alpha$ is probability under the notion of repeated sampling. If you collect data many times when the null is true, in the long run a proportion of $\alpha$ of those times you would reject. In effect it tells you the probability of a Type I error before you sample.

The p-value is instance-specific and conditional. It does not tell you the probability of a type I error, either before you sample (it can't tell you that, since it depends on the sample), or after:

If $p\geq\alpha$ then the chance you made a Type I error is zero.

If the null is true and $p<\alpha$ then the chance you made a Type I error is 1.

Take another look at the two things under discussion:

• P(Type I error) = P(reject H$_0$|H$_0$ true)

• p-value = P(sample result at least as extreme as the observed sample value|H$_0$ true, sample)

They're distinct things.

---

Edit - It appears from comments that it is necessary to address your second paragraph in detail:

The p value seems to give an exact estimate of the probability of falsely rejecting a true null hypothesis

Not so, as discussed above. (I assumed this was sufficient to render the rest of the question moot.)

α seems to be a maximum acceptable Type I error,

In effect, yes (though of course we may choose a lower $\alpha$ that the absolute maximum rate we'd be prepared to accept for a variety of reasons).

whereas p is exact.

Again, not so; it's not equivalent to $\alpha$ in the suggested sense. As I suggest, both the numerator and denominator in the conditional probability differ from the ones for $\alpha$.

Put differently it appears to give the minimum α level under which we could still reject the null.

In spite of my earlier caveats, there is a direct (and not necessarily particularly interesting) sense in which this is true. Note that $\alpha$ is chosen before the test, $p$ is observed after, so it's necessary to shift from our usual situation.

If we posit the following counterfactual:

• we have a collection of hypothesis testers, each operating at their own significance level

• they are each presented with the same set of data

then it is the case that the p-value is a dividing line between those testers that reject and those that accept. In that sense, the p-value is the minimum α level under which testers could still reject the null. But in a real testing situation, $\alpha$ is fixed, not variable, and the probability we're dealing with is either 0 or 1 (in a somewhat similar sense to the way people say "the probability that the confidence interval includes the parameter").

Our probability statements refer to repeated sampling; if we posit a collection of testers each with their individual $\alpha$, and consider only a single data set to test one, it's not clear $\alpha$ is the probability of anything in that scenario - rather, $\alpha$ represents something if we had a collection of testers and repeated sampling where the null is true - they'd each be rejecting a proportion $\alpha$ of their nulls across samples, while $p$ would represent something about each sample.

Answer 2

Your interpretation seems about correct. The caveat I would add is that $\alpha$ is an a priori decision to be made before conducting an hypothesis test. So it's no good finding that the p-value for a test statistic is, say, 0.00021, and then reporting that your test had an $\alpha$ of 0.00021; that would make $\alpha$ and p (falsely) synonymous.

Answer 3

The $p$-value is not "an exact estimate of the probability of falsely rejecting a true null hypothesis". This probability is fixed by construction of an $\alpha$-level test. Rather it is an estimate of the probability that other realisations of the experiment are more extreme than the actual realisation. Only if the present realisation belongs to the top $\alpha$ extreme realisations, we reject the null hypothesis.

But it is right that you can imagine the $p$-value to be the minimum $\alpha$, such that , if this $\alpha$ had been chosen this way, the test would be on the border of significance to insignificance for the present data.

Maybe a different explanation helps: We say that we reject the null hypothesis, iff the present outcomes can be shown to belong to the extreme $100 \alpha \%$ of possible outcomes, provided the null hypothesis holds. The $p$-value just indicates how extreme our outcomes actually are.

Answer 4

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to a conditional probability connected to a single event (the data collected). In this case the latter is the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$. And, the $p$ is not the probability of falsely rejecting the null.

Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.

Answer 5

When one calculates a $p$-value, one is actually computing a conditional probability in which the condition being assumed to be true is the null hypothesis. So in this way, the $p$-value is in a sense a quantifier of how likely we would expect to observe a sample at least as extreme as the one we saw, assuming the sample satisfies the distributional assumptions of the null hypothesis. This latter part is extremely important, because only then can we infer from the $p$-value whether or not there exists sufficient evidence to reject the null hypothesis.

If I give you a coin but tell you nothing about whether it is fair, and you toss it $100$ times and get $99$ heads and $1$ tail, you would very likely and reasonably conclude that the coin is not in fact fair. The way you would quantify this impression, as a statistician, is to first suppose that the coin is fair, and then demonstrate through the use of a binomial proportion test, that the chance that you could have gotten such an extreme result under that supposition is incredibly small--this value is the $p$-value of the test.

That said, even though that chance is astronomically small, it still isn't zero. There is a tiny, tiny possibility that a truly fair coin, tossed $100$ times, could give you $99$ heads and $1$ tail; or $99$ tails and one head; or all heads; or all tails, all due to random chance. Just because an event is rare does not mean it is impossible. Therefore, whenever we conduct a nontrivial statistical test, there is always some possibility of error. You would be quite confident that the coin isn't fair, but you could be wrong, and the probability you could be wrong in this sense is the Type I error.