49
$\begingroup$

Would like to know how confident I can be in my $\lambda$. Anyone know of a way to set upper and lower confidence levels for a Poisson distribution?

  • Observations ($n$) = 88
  • Sample mean ($\lambda$) = 47.18182

what would the 95% confidence look like for this?

$\endgroup$
1
  • 1
    $\begingroup$ You might also consider bootstrapping your estimates. Here is a short tutorial on bootstrapping. $\endgroup$ Commented Apr 30, 2013 at 14:10

5 Answers 5

36
$\begingroup$

For Poisson, the mean and the variance are both $\lambda$. If you want the confidence interval around lambda, you can calculate the standard error as $\sqrt{\lambda / n}$.

The 95-percent confidence interval is $\hat{\lambda} \pm 1.96\sqrt{\hat{\lambda} / n}$.

$\endgroup$
5
  • 46
    $\begingroup$ This is fine when $n \lambda$ is large, for then the Poisson is adequately approximated by a Normal distribution. For smallish values or higher confidence, better intervals are available. See math.mcmaster.ca/peter/s743/poissonalpha.html for two of them along with an analysis of their actual coverage. (Here, the "exact" interval is (45.7575, 48.6392), the "Pearson" interval is (45.7683, 48.639), and the Normal approximation gives (45.7467, 48.617): it's a little too low, but close enough, because $n \lambda = 4152$.) $\endgroup$
    – whuber
    Commented Sep 9, 2011 at 15:34
  • 8
    $\begingroup$ For others confused like I was: here is a description of where the 1.96 comes from. $\endgroup$
    – maddyblue
    Commented Apr 17, 2012 at 19:26
  • 2
    $\begingroup$ How did you calculate the exact interval for this problem given the information on that website given by whuber? I couldn't follow because that site seems to only indicate how to proceed when you have one sample. Maybe I'm just not understanding something simple but my distribution has a much smaller value of lambda(n) so I can't use the normal approximation and I don't know how to compute the exact value. Any help would be greatly appreciated. Thanks! $\endgroup$
    – user12849
    Commented Jul 25, 2012 at 17:59
  • $\begingroup$ Here they are using the standard deviation of the mean right? That is, SE = sig/sqrt(N) = sqrt(lam/N) ? This would make sense since the standard deviation of single values sig tells us about the likelihood of drawing random samples from the Poisson distribution, whereas the SE as defined above tells us about our confidence in lam, given the number of samples we've used to estimate it. $\endgroup$
    – Alex
    Commented Mar 13, 2019 at 17:40
  • $\begingroup$ If $n=1$ and $\lambda=2$ the left bound of this 95%-interval would be negative: $$ 2 - 1.96 \sqrt{2} = -0.7719 < \lambda < 4.7719 = 2 + 1.96 \sqrt{2} $$ Using symmetric confidence intervals for a right-skewed distribution like the poisson distribution is not a good idea. $\endgroup$
    – asmaier
    Commented Jun 13, 2023 at 8:23
28
$\begingroup$

Patil & Kulkarni (2012, "Comparison of Confidence Intervals for the Poisson Mean: Some New Aspects", REVSTAT - Statistical Journal) discuss 19 different ways to calculate a confidence interval for the mean of a Poisson distribution.

$\endgroup$
1
  • 9
    $\begingroup$ Despite the mod's notification here, I like this answer as-is, because it points out that there's less than general consensus as to how to evaluate a measured Poisson system. $\endgroup$ Commented Jul 25, 2018 at 13:27
14
$\begingroup$

In addition to the answers that others have provided, another approach to this problem is achieved through a model based approach. The central limit theorem approach is certainly valid, and the bootstrapped estimates offer a lot of protection from small sample and mode misspecification issues.

For sheer efficiency, you can get a better confidence interval for $\lambda$ by using a regression model based approach. No need to go through derivations, but a simple calculation in R goes like this:

x <- rpois(100, 14)
exp(confint(glm(x ~ 1, family=poisson)))

This is a non-symmetric interval estimate, mind you, since the natural parameter of the poisson glm is the log relative rate! This is an advantage since there is a tendency for count data to be skewed to the right.

The above approach has a formula and it is :

$$\exp\left( \log \hat{\lambda} \pm \sqrt{\frac{1}{n\hat{\lambda} }}\right)$$

This confidence interval is "efficient" in the sense that it comes from maximum likelihood estimation on the natural parameter (log) scale for Poisson data, and provides a tighter confidence interval than the one based on the count scale while maintaining the nominal 95% coverage.

$\endgroup$
7
  • 1
    $\begingroup$ +1 I think I would use a different adjective than efficiency though (or be more clear you mean computational or code golf efficiency). whuber's comment points to a resource that gives exact intervals, and the glm approach is based on asymptotic results as well. (It is more general though, so I like recommending that approach as well.) $\endgroup$
    – Andy W
    Commented Aug 8, 2014 at 23:14
  • $\begingroup$ Actually thinking about this some more, the exact coverage whuber links to (I think) is only applicable if you specify $\mu$ without looking at the data. See a quick simulation, the coverage calculated based on the observed value (for new observations) is much lower. Quick simulation here. $\endgroup$
    – Andy W
    Commented Aug 9, 2014 at 0:28
  • 2
    $\begingroup$ What is your authority for that formula. Can we have a citation? $\endgroup$
    – pauljohn32
    Commented Apr 4, 2018 at 22:20
  • $\begingroup$ @AndyW: your link is not valid for the quick simulation $\endgroup$
    – pauljohn32
    Commented Apr 4, 2018 at 22:20
  • 2
    $\begingroup$ @pauljohn32 check out Casella Berger's text especially on exponential family, the log rate is the natural parameter. $\endgroup$
    – AdamO
    Commented Apr 4, 2018 at 22:31
4
$\begingroup$

Given an observation from a Poisson distribution,

  • the number of events counted is n.
  • the mean ($\lambda$) and variance ($\sigma^2$) are equal.

Step by step,

  • The estimate for the mean is $\hat \lambda = n \approx \lambda$
  • Assuming the number of events is big enough ($n \gt 20$), the standard error is the standard deviation $\sigma$, which we can also estimate,

$$stderr = \sigma = \sqrt{\lambda} \approx \sqrt{n} $$

Now, the 95% confidence interval is,

$$ I = \hat \lambda \pm 1.96 \space stderr = n \pm 1.96 \space \sqrt{n}$$

[Edited] Some calculations based on the question data,

  • Assuming the $\lambda$ indicated in the question has been externally checked or was given to us, i.e., it is a good piece of information not an estimation.

    I am making this assumption as the original question does not provide any context about the experiment or how the data was obtained (which is of the utmost importance when manipulating statistical data).

  • The 95% confidence interval is, for the particular case,

$$ I = \lambda \pm 1.96 \space stderr = \lambda \pm 1.96 \space \sqrt{\lambda} = 47.18182 \pm 1.96 \space \sqrt{47.18182} \approx [33.72, 60.64] $$

Hence, as the measurement (n=88 events) is outside the 95% confidence interval, we conclude that,

  1. The process does not follow a Poisson process, or,

  2. The $\lambda$ we have been given is not correct.


Important note: the first accepted answer above is wrong, as it incorrectly states that the standard error for a Poisson observation is $\sqrt{\lambda/n}$. That is the standard error for a Sample Mean (Survey Sample) process.

$\endgroup$
4
  • 2
    $\begingroup$ Welcome to the site! But @Travis "would like to know how confident I can be in my $\lambda$", so it should be a confidence interval around the sample mean. Besides, what do you mean by $n\approx\lambda$, given they are 88 and 47 respectively? $\endgroup$
    – Randel
    Commented Aug 8, 2014 at 19:07
  • 2
    $\begingroup$ Thanks! I have now edited the answer including some specific calculations. The question does not explain how $\lambda$ and n have been obtained, so I made an educated guess. As you say, if n differs too much from $\lambda$ is the first hint that the model may not be Poisson or the measurement was not done right. One way to check it is precisely calculating the 95% confidence interval which, in this case, shows n is outside the interval. $\endgroup$ Commented Aug 8, 2014 at 20:53
  • 4
    $\begingroup$ I believe the response by jose.angel.jiminez above is incorrect, and arises from misreading the original question. The original poster stated "Observations (n) = 88" -- this was the number of time intervals observed, not the number of events observed overall, or per interval. The average number of events per interval, over the sample of 88 observing intervals, is the lambda given by the original poster. (I'd have included this as a comment to Jose's post, but am too new to the site to be allowed to comment.) $\endgroup$
    – hikinfiend
    Commented Mar 20, 2015 at 18:09
  • $\begingroup$ @user44436 added an answer that was supposed to be a comment. I repost it as a comment so that you can see it and because as a non-answer it may get removed: ------- I believe the response by jose above is incorrect and arises from misreading the original question. The original poster stated Observations (n) = 88 - this was the number of time intervals observed, not the number of events observed overall, or per interval. The average number of events per interval over the sample of 88 observing intervals is the lambda given by the original poster. $\endgroup$
    – Mörre
    Commented May 11, 2015 at 11:58
0
$\begingroup$

The Poisson distribution Wikipedia page gives an answer. Suppose $$ K \sim \text{Pois}(\lambda) $$ with $\lambda$ unknown and we sample $K$ $n$ times getting $k_1, k_2, \ldots, k_n$. We let $$ \begin{align*} k =& \sum_{i=1}^n k_i\\ \end{align*} $$

Suppose we want a confidence interval with confidence level $0\le c \le 1$. Let $\alpha = 1-c$. Then $$ CI_{\alpha} = \left\{\lambda: \frac{1}{n}\text{PPF}_{\chi^2_{2k}}(\alpha/2) \le \lambda \le \frac{1}{n}\text{PPF}_{\chi^2_{2k+2}}(1-\alpha/2)\right\} $$ Where $\chi^2_D$ is a $\chi^2$-distribution with $D$ degrees of freedom and $\text{PPF}_{\chi^2_D}$ is the probability point function (PPF) or quantile function for that distribution.

I've done some simulations and the coverage seems to be good as long as $k=n\mu > 1$. I haven't compared to any other methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.