# Neyman-Pearson lemma

I have read the Neyman–Pearson lemma from the book Introduction to the Theory of Statistics by Mood, Graybill and Boes. But I have not understood the lemma.

Can anyone please explain the lemma to me in plain words? What does it state?

Neyman-Pearson Lemma : Let $$X_1,\ldots,X_n$$ be a random sample from $$f(x;\theta)$$, where $$\theta$$ is one of two known values $$\theta_0$$ and $$\theta_1$$, and let $$0<\alpha<1$$ be fixed.

Let $$k^*$$ be a positive constant and $$C^*$$ be a subset of $$\mathscr X$$ which satisfy :$$\tag 1 P_{\theta_0}[(X_1,\ldots,X_n)\in C^*] = \alpha$$ $$\tag 2 \lambda=\frac{L(\theta_0;x_1,\ldots,x_n)}{L(\theta_1;x_1,\ldots,x_n)} = \frac{L_0}{L_1} \le k^*\quad \text{if } (x_1,\ldots,x_n)\in C^*$$ $$\text{and}\quad \lambda\ge\quad k^* \text{ if } (x_1,\ldots,x_n)\in \bar C^*$$ Then the test $$\gamma^*$$ corresponding to the critical region $$C^*$$ is a most powerful test of size $$\alpha$$ of $$\mathscr H_0:\theta=\theta_0$$ versus $$\mathscr H_1:\theta=\theta_1$$

Expressed in words, I have understood that the two criteria specify

(1) P[rejecting null hypothesis | null hypothesis is true] =significance level

(2) reject null hypothesis when the likelihood ratio, $$\lambda\le$$ some positive constant $$k^*$$ if $$(x_1,\ldots,x_n)$$ fall in the critical region

Then the test is the most powerful test of a simple hypothesis.

• Why is it only for simple hypotheses? Can't it be for composite hypothesis? Is my explanation in words correct?

I think you understood the lemma well.

Why it does not work for a composite alternative? As you can see in the likelihood ratio, we need to plug in the parameter(s) for the alternative hypothesis. If the alternative is composite, which parameter are you going to plug in?

• You can get it to work for composite alternatives if the likelihood ratio is monotone. – Michael R. Chernick May 2 '17 at 20:41

I recently wrote an entry in a linkedin blog stating Neyman Pearson lemma in plain words and providing an example. I found the example eye opening in the sense of providing a clear intuition on the lemma. As often in probability, it is based on a discrete probability mass function so it is easy to undertand than when working with pdf's. Also, take into account I define the likelihood ratio as the likelihood of the alternative hypothesis vs. the null hypothesis, contrary to your lemma statement. The explanation is the same, but rather than less than is now greater than. I hope it helps...

Those of you who work in data analysis and have been through some statistics courses may have come to know Neyman-Pearson lemma (NP-lemma). The message is simple, the demonstration not so much but what I always found difficult was to get a common-sense feeling of what it was about. Reading a book named "Common Errors in Statistics" by P.I.Good and J.W.Hardin I got to an explanation and example that helped me get this gut feeling about the NP-lemma I had always missed.

In not 100% mathematically perfect language, what Neyman-Pearson tells us is that the most powerful test one can come up to validate a given hypothesis within a certain significance level is given by a rejection region made by all possible observations coming from this test with a likelihood ratio above a certain threshold... woahhh! Who said it was easy!

Keep calm and deconstruct the lemma:

1. Hypothesis. In statistics one always works with two hypothesis that a statistical test should reject or not reject. There is the null hypothesis, that will not be rejected until sample evidence against it is strong enough. There is also the alternative hypothesis, the one we will take if the null seems to be false.
2. Power of a test (a.k.a. sensitivity) tells us which proportion of times we will correctly reject the null hypothesis when it is wrong. We want powerful tests, so most of the time we reject the null hypothesis we are right!
3. Significance level of a test (a.k.a. false positive rate) tells us which proportion of times we will wrongly reject the null hypothesis when it is true. We want a small significance level so most of the times we reject the null hypothesis we are not wrong!
4. Rejection region, given all possible outcomes of the test, the rejection region includes those outcomes that will make us reject the null hypothesis in benefit of its alternative one.
5. Likelihood is the probability of having seen the observed outcome of the test given that the null hypothesis (Likelihood of the null hypothesis) or the alternative one (Likelihood of the alternative hypothesis) were true.
6. Likelihood ratio, is the ratio of the alternative hypothesis likelihood divided by the null hypothesis likelihood. If the test outcome was very much expected if the null hypothesis were true versus the alternative one, the likelihood ratio should be small.

Enough definitions! (although if you look at them carefully, you will realize they are very insightful!). Let's go to what Neyman and Pearson tell us: if you want to have the best possible statistical test from the point of view of its power just define the rejection region by including those test results that have the highest likelihood ratio, and keep adding more test results until you reach a certain value for the number of times your test will reject the null hypothesis when it is true (significance level).

Let's see an example where hopefully everything will come together. The example is based on the book mentioned above. It is completely made up by myself so it should not be viewed as reflecting any reality or personal opinion.

Imagine one wants to determine whether somebody is in favor of setting immigration quotas (null hypothesis) or not (alternative hypothesis) by asking his/her feelings versus the European Union.

Imagine we knew the actual probability distribution for both types of people regarding the answer to our question:

Let's imagine we are willing to accept a false positive error of 30%, that is, 30% of the time we will reject the null hypothesis and assume the interviewed person is against quotas when he/she is really for them. How would we construct the test?

According to Neyman and Pearson we would first take the result with the highest likelihood ratio. This is the answer of "really like the EU" with a ratio of 3. With this result, if we assume somebody is against quotas when he/she said he "really likes the EU", 10% of the time we would be assigning for quotas people as against (significance). However we would only be correctly classifying against quota people 30% of the time (power) as not everybody in this group have the same opinion about the EU.

This seems to be a poor result as far as power is concerned. However, the test does not make many mistakes at misclassifying for quota people (significance) . As we are more flexible regarding significance, let's look for the next test result that we should add to the bag of answers that reject the null hypothesis (rejection region).

The next answer with the highest likelihood ratio is "like the EU". If we use the answers "really like" and "like" the EU as test results that allow us to reject the null hypothesis of somebody being for quotas, we would be misclassifying for quotas people as not 30% of the time (10% from the "really like" and 20% from the "like") and we would be correctly classifying against quotas people 65% of the time (30% from "really like" and 35% from "like"). In statistical jargon: our significance increased from 10% to 30% (bad!) while the power of our test increased from 30% to 65% (good!).

This is a situation all statistical tests have. There is not something such as a free lunch even in statistics! If you want to increase the power of your test you do it at the expense of increasing the level of significance. Or in simpler terms: you want to better classify the good guys, you will do at the expense of having more bad guys looking good!

Basically, now we are done! We created the most powerful test we could with the given data and a significance level of 30% by using "really like" and "like" labels to determine if somebody is against quotas... are we sure?

What would have happened if we had included in the second step after the "really like" answer was chosen, the answer "indifferent" instead of "like"? The significance of the test would have been the same than before at 30%: 10% for quota people answer "really" like and 20% for quota people answer "dislike". Both tests would be as bad at misclassifying for quota individuals. However, the power would get worse! With the new test we would have a power of 50% instead of the 65% we had before: 30% from "really likes" and 20% from "indifferent". With the new test we would be less precise at identifying against quota individuals!

Who helped out here? Neyman-Person likelihood ratio remarkable idea! Taking at each time the answer with the highest likelihood ratio ensured us that we include in the new test as much power as possible (large numerator) while keeping the significance under control (small denominator)!

• Wow, just seeing everything in that table helped a ton, and referring to parts of it helped a ton. Thank you! – Yatharth Agarwal Feb 13 '19 at 21:45

# The Context

(In this section I'm just going to explain hypothesis testing, type one and two errors, etc, in my own style. If you're comfortable with this material, skip to the next section)

The Neyman-Pearson lemma comes up in the problem of simple hypothesis testing. We have two different probability distributions on a common space $$\Omega$$: $$P_0$$ and $$P_1$$, called the null and the alternative hypotheses. Based on a single observation $$\omega\in\Omega$$, we have to come up with a guess for which of the two probability distributions is in effect. A test is therefore a function which to each $$\omega$$ assigns a guess of either "null hypothesis" or "alternative hypothesis". A test can obviously be identified with the region on which it returns "alternative", so we're just looking for subsets (events) of the probability space.

Typically in applications, the null hypothesis corresponds to some kind of status quo, whereas the alternative hypothesis is some new phenomenon which you're trying to prove or disprove is real. For example, you may be testing someone for psychic powers. You run the standard test with the cards with squiggly lines or what not, and get them to guess a certain number of times. The null hypothesis is that they'll get no more than one in five right (since there's five cards), the alternative hypothesis is that they're psychic and may get more right.

What we'd like to do is minimize the probability of making a mistake. Unfortunately, that's a meaningless notion. There are two ways you could make a mistake. Either the null hypothesis is true, and you sample an $$\omega$$ in your test's "alternative" region, or the alternative hypothesis is true, and you sample the "null" region. Now, if you fix a region $$A$$ of the probability space (a test), then the numbers $$P_0(A)$$ and $$P_1(A^{c})$$, the probabilities of making those two kinds of errors, are completely well-defined, but since you have no prior notion of "probability that the null/alternative hypothesis is true", you can't get a meaningful "probability of either kind of mistake". So this is a fairly typical situation in mathematics where we want the "best" of some class of objects, but when you look closely, there is no "best". In fact, what we're trying to do is minimize $$P_0(A)$$ while maximizing $$P_1(A)$$, which are clearly opposing goals.

Keeping in mind the example of the psychic abilities test, I like to refer to the type of mistake in which the null is true but you conclude the alternative as true as "delusion" (you believe the guy's psychic but he's not), and the other kind of mistake as "obliviousness".

# The Lemma

The approach of the Neyman-Pearson lemma is the following: let's just pick some maximal probability of delusion $$\alpha$$ that we're willing to tolerate, and then find the test that has minimal probability of obliviousness while satisfying that upper bound. The result is that such tests always have the form of a likelihood-ratio test:

Proposition (Neyman-Pearson lemma)

If $$L_0, L_1$$ are the likelihood functions (PDFs) of the null and alternative hypotheses, and $$\alpha > 0$$, then the region $$A\subseteq \Omega$$ which maximizes $$P_1(A)$$ while maintaining $$P_0(A)\leq \alpha$$ is of the form

$$A=\{\omega\in \Omega \mid \frac{L_1(\omega)}{L_0(\omega)} \geq K \}$$

for some constant $$K>0$$. Conversely, for any $$K$$, the above test has $$P_1(A)\geq P_1(B)$$ for any $$B$$ such that $$P_0(B)\leq P_0(A)$$.

Thus, all we have to do is find the constant $$K$$ such that $$P_0(A)=\alpha$$.

The proof on Wikipedia at time of writing is a pretty typically oracular mathematical proof that just consists in conjecturing that form and then verifying that it is indeed optimal. Of course the real mystery is where did this idea of taking a ratio of the likelihoods even came from, and the answer is: the likelihood ratio is simply the density of $$P_1$$ with respect to $$P_0$$.

If you've learned probability via the modern approach with Lebesgue integrals and what not, then you know that under fairly unrestrictive conditions, it's always possible to express one probability measure as being given by a density function with respect to another. In the conditions of the Neyman-Pearson lemma, we have two probability measures $$P_0$$, $$P_1$$ which both have densities with respect to some underlying measure, usually the counting measure on a discrete space, or the Lebesgue measure on $$\mathbb R^n$$. It turns out that since the quantity that we're interested in controlling is $$P_0(A)$$, we should be taking $$P_0$$ as our underlying measure, and viewing $$P_1$$ in terms of how it relates to $$P_0$$, thus, we consider $$P_1$$ to be given by a density function with respect to $$P_0$$.

The heart of the lemma is therefore the following:

Let $$\mu$$ be a measure on some space $$\Omega$$, and let $$f$$ be a positive, integrable function on $$\Omega$$. Let $$\alpha > 0$$. Then the set $$A$$ with $$\mu(A)\leq\alpha$$ which maximizes $$\int_A fd\mu$$ is of the form $$\{\omega\in\Omega\mid f(\omega)\geq K\}$$ for some constant $$K>0$$, and conversely, any such set maximizes $$\int f$$ over all sets $$B$$ smaller than itself in measure.

Suppose you're buying land. You can only afford $$\alpha$$ acres, but there's a utility function $$f$$ over the land, quantifying, say, potential for growing crops, and so you want a region maximizing $$\int f$$. Then the above proposition says that your best bet is to basically order the land from most useful to least useful, and buy it up in order of best to worst until you reach the maximum area $$\alpha$$. In hypothesis testing, $$\mu$$ is $$P_0$$, and $$f$$ is the density of $$P_1$$ with respect to $$P_0$$ (which, as already stated, is $$L_1/L_0$$).

Here's a quick heuristic proof: out of a given region of land $$A$$, consider some small one meter by one meter square tile, $$B$$. If you can find another tile $$B'$$ of the same area somewhere outside of $$A$$, but such that the utility of $$B'$$ is greater than that of $$B$$, then clearly $$A$$ is not optimal, since it could be improved by swapping $$B$$ for $$B'$$. Thus an optimal region must be "closed upwards", meaning if $$x\in A$$ and $$f(y)>f(x)$$, then $$y$$ must be in $$A$$, otherwise we could do better by swapping $$x$$ and $$y$$. This is equivalent to saying that $$A$$ is simply $$f^{-1}([K, +\infty))$$ for some $$K$$.