# Why is Neyman-Pearson lemma a lemma or is it a theorem?

A classical result in statistical theory is the Neyman-Pearson lemma, which not only shows the existence of tests with the most power that return a pre-specified level of Type I error, but also a way to construct such tests.

If this powerful result is a lemma, then what eventual result was this used for in the original works of Neyman & Pearson?

• an additional question may be: who attached the word 'Lemma' to the Neyman-Pearson result for the first time? Nov 24, 2022 at 8:20
• I think that in practice, it's really not relevant to wonder about a result being a "lemma" or a "theorem": every theorem is "used" in the proof in some another theorem. Moreover, it is logically always possible to reprove a lemma inside another proof, and change the phrasing in order to "hide" the "use" of a lemma.
– Plop
Nov 24, 2022 at 17:36

As Thomas Lumley asserted, Neyman and Pearson in $$\rm [I]$$ didn't mention lemma. They frequently used the word principle, basis while deducing the critical regions in various cases.

When was the first time it was marked as a lemma?

$$\bullet$$ Wilks in his book did outline the theory but again refrained from calling it as lemma.

$$\bullet$$ Cramér in his book never mentioned any lemma but explained the "basic idea of Neyman-Pearson theory".

$$\bullet$$ Lehmann termed it while "formaliz[ing] in the following theorem, the fundamental lemma of Neyman and Pearson".

$$\bullet$$ Kendall & Stuart did use the term while writing "the examples we have given so far of the use of the Neyman-Pearson Lemma ..." the "lemma due to Neyman and Pearson ..."

$$\bullet$$ In $$\rm [VI],$$ the authors detailed a Lemma, which would be the more familiar Generalized NP Lemma we are acquainted with.

Again with absolute certainty, I cannot ascertain whether this was the first time the word was introduced. But as of now, it seems.

## References:

$$\rm [I]$$ Neyman, J., & Pearson, E. S. ($$1933$$). On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, $$231(694-706), ~289–337.$$ doi:10.1098/rsta.1933.0009

$$\rm [II]$$ Mathematical Statistics, S. S. Wilks, Princeton University Press, $$1943,$$ sec. $$7.3,$$ p. $$152.$$

$$\rm [III]$$ Mathematical Methods of Satistics, Harald Cramér, Princeton University Press, $$1946,$$ sec. $$35.1,$$ p. $$527.$$

$$\rm [IV]$$ Testing Statistical Hypotheses, E. L. Lehmann, John Wiley & Sons, $$1959,$$ sec. $$3.2,$$ p. $$64.$$

$$\rm [V]$$ The Advanced Theory of Statistics: Inference and Relationship, Maurice G. Kendall, Alan Stuart, Hafner Publishing Company, $$1961,$$ sec. $$22.10,$$ p. $$166.$$

$$\rm [VI]$$ Statistical Research Memoirs: Volume $$1,$$ University College, London, Department of Statistics, $$1936,$$ p. $$11.$$

• (+1) Didn't know that the first edition of Lehmann was published in the 50'. Nov 24, 2022 at 8:47
• @utobi While I have the latest edition, and the first edition's publication date was mentioned, I saw it in the library few days ago. Nov 24, 2022 at 8:49
• one of the oldies but goodies. Nov 24, 2022 at 8:50

What people often describe as the Neyman Pearson lemma is a result proven by the lemma but not the lemma itself. The description of the lemma by cross validated is for instance:

A theorem stating that likelihood ratio test is the most powerful test of point null hypothesis against point alternative hypothesis

However, the lemma is a more abstract and technical underlying result.

The region $$\omega$$ maximises $$\int_{w \in \omega} g(w) dw$$ subject to the constraints $$\int_{w \in \omega} f_i(w) dw = c_i$$, if and only if it exists and if for some constants $$k_i$$ we have that $$g_i > \sum k_i f_i$$ everywhere inside the region and $$g_i < \sum k_i f_i$$ everywhere outside the region.

(You can see it as a sort of equivalent to Lagrange multipliers)

This Lemma is not only applied to the case of likelihood ratio tests but also to find optimal (most powerful) critical regions of other types.

### The 1933 version

Probably the first occurrence is in 1933. A proof for the lemma is given but it is not yet explicitly named as a lemma or theorem.

Neyman, Jerzy, and Egon Sharpe Pearson. "IX. On the problem of the most efficient tests of statistical hypotheses." Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 231.694-706 (1933): 289-337.

https://doi.org/10.1098/rsta.1933.0009

### The 1935 version

An early reference to the lemma occurs in 1935. It is in French and doesn't yet speak of a lemma, but it speaks of a general result (résultat général). It is actually not referencing the 1933 article, and it is referencing the future article in 1936 (with the remark 'in press')

Neyman, Jerzy. "Sur la vérification des hypothèses statistiques composées." Bulletin de la Société Mathématique de France 63 (1935): 246-266.

Ce problème peut être résolu en appliquant le résultat général que voici.

### The 1936 version

In 1936 there is reference to the proposition that explicitly calls it a lemma

London (England). et al. Statistical Research Memoirs. 1936.

### The 1939 version

In 1939 Wald speaks about a general principle in a theory. Here it is not anymore about a lemma used in a theorem. Possibly this could be the start where the meaning of the lemma started to switch to the statement about the optimal power for likelihood ratio tests?

Wald, Abraham. "Contributions to the theory of statistical estimation and testing hypotheses." The Annals of Mathematical Statistics 10.4 (1939): 299-326.

In the Neyman-Pearson theory two types of hypotheses are considered. Let $$\theta =\theta_1$$ be the hypothesis to be tested, where $$\theta_1$$ denotes a certain point if the parameter space. Denote this hypothesis by $$H_1$$ and the hypothesis $$\theta \neq \theta_1$$ by $$\bar{H}$$. The type I error is that which is made by rejecting $$H_1$$ when it is true. The type II error is that which is made by accepting $$H_1$$ when it is false. The fundamental principle in the Neyman-Pearson theory can be formulated as follows: Among all critical regions (regions of rejection of $$H_1$$, i.e. regions of acceptance of $$\bar{H}$$) for which the probability of type I error is equal to a given constant $$\alpha$$, we have to choose that region for which the probability of type II error is a minimum.

• Both your and User1865345's answers are excellent! Nov 28, 2022 at 7:21