Likelihood Ratio vs Modified Frequentist Approach (CLs)

I'm a physicist trying to finally get a hold on practical statistics for particle physics and am having problem with the following -- I apologize for the lack of formality below.

Suppose the number of events within a single channel is governed by a Poisson distribution $$P(N,\mu)$$, whose parameter for Null ($$\mu_b$$) and Alternative ($$\mu_s$$) hypotheses I obtain from external reasoning/simulations (notice that $$\mu_s \ge \mu_b$$).

I want to find a threshold for exclusion of signal at 95% CLs (not CL), which is defined by

$$CL_s = \frac{CL(\text{alternative})}{CL(\text{null})}.$$

Since I want to rule out the alternative (and $$\mu_s \ge \mu_b$$), I want to define the $$N_0$$ threshold where

$$CL_s = \frac{P(N\le N_0\mid\mu_s)}{P(N\le N_0\mid\mu_b)}\le 0.05.$$

Lack of precision aside, I understand what is happening here. However, some texts for statistics in High Energy Physics seem to suggest using the Likelihood Ratio test statistic for CLs. In this case, I try to replace $$N$$ by

$$Q \equiv -2\ln \frac{P(N\mid\mu_s)}{P(N\mid\mu_b)}$$

and then define the threshold as

$$CL_s =\frac{P(Q> Q_0\mid\mu_s)}{P(Q> Q_0 \mid \mu_b)} = \frac{1-P(Q\le Q_0 \mid \mu_s)}{1-P(Q\le Q_0 \mid \mu_b)}\le 0.05.$$

In the end, however, I end up inverting $$Q_0 = Q(N_0)$$, to obtain a limit on $$N$$.

Does this make sense at all? Do I gain something going over to $$Q$$, or is the Likelihood Ratio Test something entirely different?

Edit: For context and confirmation, check the following didactic article.

• A quick question on your notation: what stands "CL" in "CL(alternative)" and "CL(null)" for? (I can see your definition in the second equation, but am curious about the acronym, and terminology can vary from discipline to discipline)
– Ute
Commented May 31, 2023 at 16:22
• @Ute unfortunately I can't answer your question precisely, as it is something that bugs me too! In physics, lecture notes authors seem to use the CL and p-value semantics per taste: I've seen authors seek (in the same context) both CL=0.95 and CL=0.05. But I believe the most common to be CL=0.95, so that in Eq. (2) I should probably say something like $1 - CL_s = \frac{P(N\le N_0|\mu_s)}{P(N\le N_0|\mu_b)}\le 0.05.$. Commented May 31, 2023 at 16:51
• If the values are CL=0.95 and CL=0.05, then I'm pretty sure it means "Confidence Level", as I know it from statistics, too. It could be that "confidence level" in physics lecture notes is used synonymously with "rejection rate" of a test, here of $\mu= \mu_b$ versus alternative $\mu > \mu_b$. Is $CL_s$ and your formula also taken from physics lecture notes?
– Ute
Commented May 31, 2023 at 17:01
• @Ute, oh, it is definitely 'Confidence Level'! I misunderstood your question, sorry. $CL_s$ is more exactly defined as $CL_s = CL_{s+b}/CL_{b}$, usually, where the numerator corresponds to the usual confidence level of the signal+background hypothesis and the denominator to the background-only hypothesis. Commented May 31, 2023 at 17:12
• The post seems to be mixing the ideas of the likelihood ratio test and likelihoodist inference such as Richard Royall has written so well about. Commented May 31, 2023 at 18:24

If I'm understanding you correctly, $$P(N\mid \mu)$$ means the value at $$N$$ of the Poisson probability mass function with expectation $$\mu.$$

To find this, you need the value of $$N.$$ If you observe the value of $$N$$ you can find $$Q.$$ The quantity you called $$\mathrm{CL}_\mathrm{s}$$ can be used if you know $$N\le N_0$$ but you don't know $$N.$$

If you use only the fact that you know that $$N\le N_0,$$ rather than using all the relevant information that you have (i.e. the value of $$N$$) then the test will have less power. The power of a statistical test is the probability of rejecting the null hypothesis, as a function of the unobservable parameters of interest. With the less powerful test, you are less likely to reject the null hypothesis when it is false.

I think the way I've seen it written is $$Q = -\ln\frac{P(N\mid\mu_b)}{P(N\mid\mu_s)},$$ i.e. the alternative hypothesis corresponds to the denominator, so that when $$Q$$ is too big you reject the null hypothesis. It is too big when it is more than a critical value $$c$$, so chosen that $$\Pr(Q>c\mid\text{null hypothesis}) \le \alpha$$ where $$\alpha$$ is the level of the test. The level is the highest probability of a Type I error that you are willing to allow. (Type I error$${}={}$$false positive, i.e. erroneously rejecting the null hypothesis.) The choice of $$\alpha$$ is a subjective economic decision. (In this case "subjective" would mean any objectivity in its choice is located elsewhere than in the theory of statistics.)

The test using this logarithm is of course exactly equivalent to the test using the ratio $$\frac{P(N\mid\mu_b)}{P(N\mid\mu_s)}.$$ The reason for taking the logarithm is $$(1)$$ in some commonplace problems $$Q$$ has a very familiar probability distribution, and $$(2)$$ often this transforms the probability distribution of the test statistic to one that asymptotically has a chi-square distribution. "Asymptotically" means a limit is taken as the sample size increases. If the sample consists of $$n$$ independent observations $$N_1,\ldots,N_n,$$ each having a Poisson distribution, and you can base your test on $$N=N_1+\cdots+N_n,$$ which also has a Poisson distribution, and $$n$$ is the sample size. If your Poisson-distributed random variable $$N$$ is the number of raindrops the fell on a hectare of land, during a particular second of very light rain, then expanding that hectare to a larger plot of ground would be what increasing the sample size consists of. In the case of the sum $$N=N_1+\cdots+N_n,$$ the conditional probability distribution of $$(N_1,\ldots,N_n)$$ given the value of $$N$$ does not depend on $$\mu,$$ so the sum, $$N,$$ gives you all information that is relevant, as long as you know that the Poisson distribution is the right model. If you were sure of the independence of $$N_1,\ldots,N_n,$$ then knowing the whole tuple $$(N_1,\ldots,N_n),$$ rather than knowing only $$N,$$ would be relevant to judging whether the data are consistent with the Poisson distribution being the right model.

• In fact, the context is always that of defining a critical condition (such as that, informally speaking, the signal is rejected with 95% confidence) and using it to find $N_0$. I understand what you said! My question is: are the CLs and Likelihood Ratio Test used together or are they different approaches? Do I gain something using the Q-test statistic instead of N directly? Commented May 31, 2023 at 19:28
• I added an article to the question for context. Commented May 31, 2023 at 20:23
• @GaloisFan : You gain something by using the observed value of $N$ rather than only the observed fact that $N\le N_0.$ But using $N$ is equivalent to using any monotone function of $N$ in that it's the same test: the test using $N$ rejects the null hypothesis if, and only if, the test using some other monotone function of $N$ rejects the null hypothesis. What you may gain by using some other monotone function of $N$ than $N$ itself, is that the other monotone function of $N$ may be one that you know how to work with. $\qquad$ Commented Jun 2, 2023 at 16:53
• $\ldots\,$ and the reason for using $-2\log\dfrac{P(N\mid \mu_b)}{P(N\mid\mu_s)}$ is often that its distribution, in the limit as the sample size grows, is a chi-square distribution. Commented Jun 2, 2023 at 16:58
• I'm familiar with Wilk's Theorem -- but it doesn't even apply since hypotheses are usually not nested in the Particle Physics context. But thank you for your remarks, it helped a lot! Commented Jun 4, 2023 at 5:32