In a Frequentist setting, how are we able to condition on the null hypothesis being True/False?

Question

Paraphrasing Casella and Berger (2002): A hypothesis test is defined by a null hypothesis $H_0: \theta \in \Theta_0 $ and an alternative hypothesis $H_1: \theta \in \Theta_0^c = \bar{H_0}$, where $\Theta$ is a set of possible values of the population parameter $\theta$, and $\Theta_0^c = \Theta - \Theta_0 $.

In the Frequentist setting, $\theta$ is an unknown but fixed quantity (meaning, it is non-random). As a result, for a given set $\Theta_0$, the null hypothesis $H_0$ is strictly True or False (but we do not know which). The alternative hypothesis, being the negation, is correspondingly False or True.

Let our test statistic be $\hat{\theta} = f(X_1, ..., X_n)$. We define the rejection region $RR \subset Range(\hat{\theta})$ as the set of values of the test statistic $\hat{\theta}$ for which we choose to reject $H_0$. As our sample $X_1, ..., X_n$ is random, $\hat{\theta}$ is a random variable, and so the quantity $p(\hat{\theta} \in RR)$ is a valid probability, similar to simpler probabilities like $p(X \le 5)$

So far so good. However, my confusion begins when we start to define Type I errors. A Type I error is when we reject $H_0$ even though it is True. This is defined mathematically as:

$$ \alpha = p(\hat{\theta} \in RR | H_0 \text{ is True}) = p(\hat{\theta} \in RR | \theta \in \Theta_0) $$

This probability makes no sense to me. $\theta$ is not a random variable, so $\theta \in \Theta_0$ is not a probabilistic event, it is a True-or-False value. The framework of probability (as far as I understand it) does not allow us to meaningfully condition on real values or booleans, only on events.

What is going on here? The only meaningful interpretation I can make is if $\theta$ is a random variable, which violates the core idea of Frequentist statistics.

Answer 1

You're right that the notation of the form $P(X|H_0)$ is misleading if this is supposed to be a frequency distribution for $X$ under a particular model assumption. I have seen people who know what they're talking about use it as shorthand, but I've also seen people sleepwalk into using it without fully appreciating what's going on.

If we're being careful, we will use different notation. In grad school, I think we used the notation $P(X;H_0)$. This is not written as a conditional probability, and is not meant to be interpreted as one. Rather, it should be read as something like "the probability (density) of $X$ under the probability model parameterized according to $H_0$." You'll notice that Casella and Berger, who you cite, are also careful and use notation such as $P_{\theta_0}(X)$ (where $H_0$ is such that $\theta=\theta_0$; page 388).

Answer 2

The issue here seems to be more about simplified notation than a conceptual error. Let’s break it down in more detail:

You wrote:
$$ P(\hat{\theta} \in RR \mid \theta \in \Theta_0) = \alpha $$

This expression doesn't make sense when $\Theta_0$ contains more than one element, i.e., when you have a composite null hypothesis. In this case, you are not specifying a single probability distribution, as $\theta$ can take multiple values within $\Theta_0$.

Case 1: Simple Null Hypothesis

When the null hypothesis $H_0$ is simple, for example, $\Theta_0 = \{\theta_0\}$, the computation of $\alpha$ is straightforward and can be expressed as:

$$ P_{\theta_0}(\hat{\theta} \in RR) = \alpha, $$

Here, $\alpha$ represents the probability that the statistic $\hat{\theta}$ falls within the rejection region $RR$, given that the true value of $\theta$ is $\theta_0$. The subscript in $P_{\theta_0}$ indicates that the probability is calculated under the assumption that $\theta = \theta_0$, and the distribution is fully specified.

Case 2: Composite Null Hypothesis

When $\Theta_0$ contains multiple possible values, as in a composite null hypothesis, there isn’t a single $\theta$ value tied to $H_0$. This makes the interpretation of $\alpha$ more complex. In this case, the significance level is typically defined as the supremum of the rejection probability over all values of $\theta$ in $\Theta_0$:

$$ \alpha = \sup_{\theta \in \Theta_0} P_{\theta}(\hat{\theta} \in RR). $$

This ensures that $\alpha$ represents the maximum Type I error rate, regardless of the specific value of $\theta$ within $\Theta_0$. In other words, this definition guarantees that the significance level does not exceed $\alpha$ for any parameter value allowed under the null hypothesis.