Upper confidence bound for Poisson process rate parameter

Question

I am interested in computing an upper confidence bound for the rate parameter, $\lambda$, in a Poisson process. Specifically, I have a set of observations $$ X_\text{obs} = \{(n_1,t_1), \ldots, (n_N,t_N)\} $$ where $n_k$ is the number of successes in the $k$'th trial which ran for a duration of length $t_k$.

For a given a confidence level $1 - \alpha$, I assume that what I want to do is solve for $$ \lambda_{\text{upper}} = \max \{ \lambda : P(X_\text{obs} ; \lambda) \ge \alpha \}. $$ However, this doc http://faculty.washington.edu/fscholz/Reports/poissonconfbd6.pdf suggests that I want to find something like $$ \lambda_{\text{upper}} = \max \{ \lambda : P(x \le X_\text{obs} ; \lambda) \ge \alpha \} $$ for all datasets "$x$" whose sufficient statistic is less than or equal to the sufficient statistic of the observed data.

Specifically, equation (1) in the doc sums over possible outcomes, $i = 0 \ldots k$, less than or equal to the observed number of successes, $k$. And it doesn't just compute the largest $\lambda$ for which the likelihood of the observed data is at least $\alpha$.

What is the correct way to define the upper confidence bound so that I can solve for it (either in closed form, or by numerical methods)?

Btw, I am aware that $\lambda$ is the mean number of counts per period, and so one could use deviation bounds based on the CLT to derive a UCB. However, I will be dealing with sparse counts (including zeros), so I am not sure that using deviation bounds will give me good performance. And regardless, I would like to see how to derive the UCB using the definition and/or first principles.

Answer 1

As you notice, you can't have proper confidence bound with only the central limit theorem. In fact, confidence bounds are finite time version of the CLT.

Poisson random variables are not subgaussian, hence we don't have the classic Hoeffding bound that enjoys other subgaussian random variable (like gaussians, or bounded r.v.). However, they are sub-exponential and enjoy similar bounds. Sub-exponential have stronger tail than subgaussian (the probability of "extreme" events decay at a slower rate with the "extremeness" of the event) and this is why they need other concentration inequalities (with slower rate). This may be related to your intuition about "sparse count".

You may want to look to the detail here. To learn more about CLT and concentration result for subgaussian, you can look to chapter 5 of this book.

Answer 2

Confidence intervals are frequentists tools. From a frequentist point of view, there is no such thing as confidence on the parameter. There is only one true parameter $\lambda$ which specifies your random process. This random process generates a sequence of data. Frequentists say that in other realities, the same process has generated different sequences of data. You can weight each reality with a probability. Then, you can add the weights of "realities" which have the same average. It gives you the probability of an average, i.e. the weight of realities which have this average.

The CLT tells you that the probability distribution over these averages of sequences tends to be a gaussian with a mean equals to the mean of your random process and a shriking variance as the size of the sequence grows. Confidence Intervals are tools for finite sequences. Yet, with the CLT, we understand that your $P\left(X_{\mathrm{obs}} ; \lambda\right) \geq \alpha$ does not make sense : the probability that a (almost) gaussian to take one specific value is (almost) zero. A finite probability $\alpha$ (or $1-\alpha$) can only be obtained by considering an interval of value.

Hence, CIs weight the realities which have an average which deviates by a certain quantity from the mean of your distribution. This is why we have $P\left(x \leq X_{\mathrm{obs}} ; \lambda\right) \geq \alpha$. For poisson process, the mean is your parameter $\lambda$. The average is an empirical estimate of this $\lambda$. CIs quantify how likely is the deviation between the two.

CIs involve the size of the sample $N$, the probability threshold $1 - \alpha$, and the deviation $\Delta = \hat{\lambda} - \lambda$. For a given probability $1 - \alpha$ and a sample size $N$, you can find the corresponding ""maximal"" deviation (e.g. in the link I sent you). In fine, it meant : "there is a fraction $1-\alpha$ of "realities" for which the empirical mean is at most at a $\Delta$ distance of the true parameter $\lambda$. Hence, there is no maximum on parameter (again, there is only one parameter) in confidence intervals.

I think you may be interested by Credible Intervals which is the Bayesian equivalent (there is only one reality with true data, and multiple more or less credible generating processes). The two are often very similar in shape but differs in nature/ philosophical perspective.