I already know the definition of an "unbiased" hypothesis test: $$\max\{\pi(\theta) : \theta \in \Omega_0\} \leq \min\{\pi(\theta) : \theta \in \Omega - \Omega_0\},$$ where $\pi(\theta)$ is a power function for the test. However, unlike an unbiased estimator, which is quite intuitive, I can't understand the actual meaning of an "unbiased" test. I found this related question (Unbiased test, what does it mean actually?), but I still need some help. Why do we call a test "unbiased" if it satisfies the above condition?


Some preliminary historical information

This is an interesting question, and one can find the answer by doing a bit of detective work on the original meanings of "unbiasedness" for different kinds of statistical problems. In the early 20th century, the classical statisticians developed a number of classical statistical methods, and they also formulated some conditions in different statistical contexts that they considered favourable conditions. They referred to each of these conditions as "unbiasedness", even though they were different conditions arising in different kinds of statistical problems.

Jerzy Neyman and Karl Pearson developed the condition of an "unbiased" hypothesis test, Neyman developed the condition for an "unbiased" confidence interval, and Florence David and Neyman developed the condition of an "unbiased" estimator. By the late 1930s and throughout the 1940s there were these three different concepts that were all called "unbiasedness" that operated in different contexts. Neyman had a hand in formulating each of these conditions, and from the fact that they were all given the same name, we can see that he clearly felt that these disparate conditions were essentially expressing the same statistical property.

In the late 1940s, the statistician Erich Lehmann took on the problem of investigating and unifying these disparate conditions using statistical decision theory. This problem was addressed in his seminal paper, Lehmann (1951) (full citation below). This paper presents a unified theory of unbiasedness grounded in statistical decision theory (which is sometimes called "Lehmann-unbiasedness" or "L-unbiasedness" after Lehmann). Lehmann shows that a single decision-theoretic condition for "unbiasedness" subsumes the three specific conditions used in hypothesis testing, confidence intervals, and point estimation.

Aside from being a brilliant paper in its own right, this paper also demonstrates the incredible genius of the early classical statisticians (particularly Jerzy Neyman), who were able to formulate useful statistical conditions for different types of problems, and call them all "unbiasedness" even though they had not yet developed a unifying theory for this. These early statisticians were able to see intuitively that all these seemingly disparate concepts related to the same underlying inchoate concept of "bias", even though the unifying concept had not yet been formulated. Lehmann came along and formalised this in his paper, and showed that these early statisticians had named these concepts appropriately, in a way that could be unified within one broader definition.

An outline of the theory of "L-unbiasedness" ("Lehmann-unbiasedness")

If you have the mathematical background to do so, I strongly recommend you read Lehmann's paper for a full exposition of the theory (see also Section 1.5 of his book on hypothesis testing). His concept of unbiasedness is grounded in statistical decision theory. Suppose you observe a data vector $\mathbf{x} \in \mathscr{X}$ from a model parameterised by a parameter $\theta \in \Theta$. Suppose you have a decision procedure $\delta: \mathscr{X} \rightarrow \mathscr{D}$ mapping each possible observed data vector to a decision in a set $\mathscr{D}$, and a loss function $L: \Theta \times \mathscr{D} \rightarrow \mathbb{R}_+$ giving a loss that depends on the parameter value and the decision made.

Now, suppose that for each parameter $\theta \in \Theta$ there is a unique correct decision $d(\theta) \in \mathscr{D}$ and each decision in the decision set is correct for some parameter value. Suppose also that for any decision, the loss is invariant over parameter values for which that is the correct decision. In that case the loss depends only on the decision $\delta(\mathbf{x})$ which was taken, and the correct decision $d(\theta)$. Suppose we now denote this loss by $\tilde{L}(d(\theta), \delta(\mathbf{x}))$. Within this decision-theoretic context, Lehmann says that the decision procedure $\delta$ is "L-unbiased" if for all $\theta \in \Theta$ we have:

$$\mathbb{E}_\theta[ \tilde{L}(d(\theta), \delta(\mathbf{X}))] = \min_{d' \in \mathscr{D}} \mathbb{E}_\theta[ \tilde{L}(d', \delta(\mathbf{X}))].$$

This condition says that, if $\theta$ is the true parameter value, then the expected loss is minimised when the decision procedure selects the correct decision associated with that parameter. A decision procedure that does this is "L-unbiased" and a decision procedure that fails to do this is "L-biased".

In his paper, Lehmann shows that this concept of unbiasedness reduces to the specific forms of "unbiasedness" in hypothesis tests, confidence intervals, and point estimation, under some simple and compelling forms for the loss function. For point estimation "L-unbiasedness" reduces to the standard concept of unbiasedness under squared-error loss for the estimator. For confidence intervals "L-unbiasedness" reduces to the standard concept of unbiasedness using fixed loss for exclusion of the parameter from the interval (and zero loss otherwise). In hypothesis testing "L-unbiasedness" reduces to the standard concept of unbiasedness under the loss function described below.

In hypothesis testing, Lehmann considered the decisions $d_0$ and $d_1$ to accept or reject the null hypothesis, and uses a loss function that has zero loss for a correct decision and fixed non-zero loss for an incorrect decision. (The loss for a Type I error may be different to the loss for a Type II error, but the losses are fixed over parameter values within the same hypotheses.) This gives the loss function:

$$L(\theta, d) = \begin{cases} L_\text{I} \cdot \mathbb{I}(d=d_1) & & & \text{if } \theta \in \Theta_0, \\[6pt] L_\text{II} \cdot \mathbb{I}(d=d_0) & & & \text{if } \theta \in \Theta_1, \\[6pt] \end{cases}$$

where $\Theta_0$ and $\Theta_1$ denote the null and alternative parameter spaces respectively, and $L_\text{I}>0$ and $L_\text{II}>0$ are the losses for Type I and Type II errors respectively. In this case the condition for L-unbiasedness reduces to:

$$\begin{align} \mathbb{P}_\theta(\delta(\mathbf{X}) = d_1) &\geqslant \frac{L_\text{I}}{L_\text{I}+L_\text{II}} \quad \quad \quad \text{for } \theta \in \Theta_0, \\[6pt] \mathbb{P}_\theta(\delta(\mathbf{X}) = d_1) &\leqslant \frac{L_\text{I}}{L_\text{I}+L_\text{II}} \quad \quad \quad \text{for } \theta \in \Theta_1. \\[6pt] \end{align}$$

This is of course the definition of an unbiased hypothesis test, taking $\alpha = L_\text{I}/(L_\text{I}+L_\text{II})$. You can read more detail and more interesting discussion in the Lehmann paper, but this gives you the essentials of his basic concept and how it reduces to the concept used in the context of hypothesis testing.

Lehmann, E. L. (1951) A general concept of unbiasedness. Annals of Mathematical Statistics 22(4), pp. 587-592.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.