Determining the sample size for a counting experiment In a simple hypothesis testing, the sample size N is fixed in advance of the experiment. However, in a counting experiment in particle physics, people often define the observation time within which the data are collected. If we do a likelihood ratio test for such an experiment, I think we should treat the sample size as a Poisson random variable with the mean value that is the product of the expected event rate and the observation time. However, I see that in some particle physics experiment N is set to the size of the actual data sample. How is this approach justified?
 A: The answer might depend on what hypothesis test you are doing.
Some things can be justified by the
Law of Total Expectation.
For example, if you have a test statistic, $T$ and you have a test procedure that rejects the null hypothesis if $T>c_n$ and you know that conditional on $N=n$, the procedure has a type 1 error rate of $P[T>c_n|\theta=0,N=n]=0.05$, then the unconditional type 1 error rate is also 5%.
$$E[E[I(T>c_n)|\theta=0,N=n]]=E[0.05]=0.05$$
In the inner expected value, it is integrating over the distribution of $T$ and in the outer expected value, it is averaging over the distribution of $N$.
The power should be different for each $N$, but if you can find the power for each $n$ , then you can use that formula to find the expected value of the conditional power- that will be the unconditional power.
If I have a test statistic like $\frac{\bar{X}-\mu}{\sqrt{var}}$ where $var$ is the estimated variance, I wondered whether it matters whether I use the conditional (on $N$) estimated variance or unconditional estimated variance.  It turns out not to matter in terms of the power or type 1 error rate if the data are normally distributed with sample size N where N is POssion having mean anywhere from 25 to 10,000 (the scenarios I tried).
