33
$\begingroup$

Would like to know how confident I can be in my $\lambda$. Anyone know of a way to set upper and lower confidence levels for a Poisson distribution?

  • Observations ($n$) = 88
  • Sample mean ($\lambda$) = 47.18182

what would the 95% confidence look like for this?

$\endgroup$
  • $\begingroup$ You might also consider bootstrapping your estimates. Here is a short tutorial on bootstrapping. $\endgroup$ – Mark T Patterson Apr 30 '13 at 14:10
28
$\begingroup$

For Poisson, the mean and the variance are both $\lambda$. If you want the confidence interval around lambda, you can calculate the standard error as $\sqrt{\lambda / n}$.

The 95-percent confidence interval is $\hat{\lambda} \pm 1.96\sqrt{\hat{\lambda} / n}$.

$\endgroup$
  • 27
    $\begingroup$ This is fine when $n \lambda$ is large, for then the Poisson is adequately approximated by a Normal distribution. For smallish values or higher confidence, better intervals are available. See math.mcmaster.ca/peter/s743/poissonalpha.html for two of them along with an analysis of their actual coverage. (Here, the "exact" interval is (45.7575, 48.6392), the "Pearson" interval is (45.7683, 48.639), and the Normal approximation gives (45.7467, 48.617): it's a little too low, but close enough, because $n \lambda = 4152$.) $\endgroup$ – whuber Sep 9 '11 at 15:34
  • 6
    $\begingroup$ For others confused like I was: here is a description of where the 1.96 comes from. $\endgroup$ – mjibson Apr 17 '12 at 19:26
  • 2
    $\begingroup$ How did you calculate the exact interval for this problem given the information on that website given by whuber? I couldn't follow because that site seems to only indicate how to proceed when you have one sample. Maybe I'm just not understanding something simple but my distribution has a much smaller value of lambda(n) so I can't use the normal approximation and I don't know how to compute the exact value. Any help would be greatly appreciated. Thanks! $\endgroup$ – user12849 Jul 25 '12 at 17:59
  • $\begingroup$ Here they are using the standard deviation of the mean right? That is, SE = sig/sqrt(N) = sqrt(lam/N) ? This would make sense since the standard deviation of single values sig tells us about the likelihood of drawing random samples from the Poisson distribution, whereas the SE as defined above tells us about our confidence in lam, given the number of samples we've used to estimate it. $\endgroup$ – AlexG Mar 13 '19 at 17:40
18
$\begingroup$

This paper discusses 19 different ways to calculate a confidence interval for the mean of a Poisson distribution.

http://www.ine.pt/revstat/pdf/rs120203.pdf

$\endgroup$
  • 2
    $\begingroup$ Despite the mod's notification here, I like this answer as-is, because it points out that there's less than general consensus as to how to evaluate a measured Poisson system. $\endgroup$ – Carl Witthoft Jul 25 '18 at 13:27
7
$\begingroup$

In addition to the answers that others have provided, another approach to this problem is achieved through a model based approach. The central limit theorem approach is certainly valid, and the bootstrapped estimates offer a lot of protection from small sample and mode misspecification issues.

For sheer efficiency, you can get a better confidence interval for $\lambda$ by using a regression model based approach. No need to go through derivations, but a simple calculation in R goes like this:

x <- rpois(100, 14)
exp(confint(glm(x ~ 1, family=poisson)))

This is a non-symmetric interval estimate, mind you, since the natural parameter of the poisson glm is the log relative rate! This is an advantage since there is a tendency for count data to be skewed to the right.

The above approach has a formula and it is :

$$\exp\left( \log \hat{\lambda} \pm \sqrt{\frac{1}{n\hat{\lambda} }}\right)$$

This confidence interval is "efficient" in the sense that it comes from maximum likelihood estimation on the natural parameter (log) scale for Poisson data, and provides a tighter confidence interval than the one based on the count scale while maintaining the nominal 95% coverage.

$\endgroup$
  • $\begingroup$ +1 I think I would use a different adjective than efficiency though (or be more clear you mean computational or code golf efficiency). whuber's comment points to a resource that gives exact intervals, and the glm approach is based on asymptotic results as well. (It is more general though, so I like recommending that approach as well.) $\endgroup$ – Andy W Aug 8 '14 at 23:14
  • $\begingroup$ Actually thinking about this some more, the exact coverage whuber links to (I think) is only applicable if you specify $\mu$ without looking at the data. See a quick simulation, the coverage calculated based on the observed value (for new observations) is much lower. Quick simulation here. $\endgroup$ – Andy W Aug 9 '14 at 0:28
  • 1
    $\begingroup$ What is your authority for that formula. Can we have a citation? $\endgroup$ – pauljohn32 Apr 4 '18 at 22:20
  • $\begingroup$ @AndyW: your link is not valid for the quick simulation $\endgroup$ – pauljohn32 Apr 4 '18 at 22:20
  • 1
    $\begingroup$ @pauljohn32 check out Casella Berger's text especially on exponential family, the log rate is the natural parameter. $\endgroup$ – AdamO Apr 4 '18 at 22:31
5
$\begingroup$

Given an observation from a Poisson distribution,

  • the number of events counted is n.
  • the mean ($\lambda$) and variance ($\sigma^2$) are equal.

Step by step,

  • The estimate for the mean is $\hat \lambda = n \approx \lambda$
  • Assuming the number of events is big enough ($n \gt 20$), the standard error is the standard deviation $\sigma$, which we can also estimate,

$$stderr = \sigma = \sqrt{\lambda} \approx \sqrt{n} $$

Now, the 95% confidence interval is,

$$ I = \hat \lambda \pm 1.96 \space stderr = n \pm 1.96 \space \sqrt{n}$$

[Edited] Some calculations based on the question data,

  • Assuming the $\lambda$ indicated in the question has been externally checked or was given to us, i.e., it is a good piece of information not an estimation.

    I am making this assumption as the original question does not provide any context about the experiment or how the data was obtained (which is of the utmost importance when manipulating statistical data).

  • The 95% confidence interval is, for the particular case,

$$ I = \lambda \pm 1.96 \space stderr = \lambda \pm 1.96 \space \sqrt{\lambda} = 47.18182 \pm 1.96 \space \sqrt{47.18182} \approx [33.72, 60.64] $$

Hence, as the measurement (n=88 events) is outside the 95% confidence interval, we conclude that,

  1. The process does not follow a Poisson process, or,

  2. The $\lambda$ we have been given is not correct.


Important note: the first accepted answer above is wrong, as it incorrectly states that the standard error for a Poisson observation is $\sqrt{\lambda/n}$. That is the standard error for a Sample Mean (Survey Sample) process.

$\endgroup$
  • 1
    $\begingroup$ Welcome to the site! But @Travis "would like to know how confident I can be in my $\lambda$", so it should be a confidence interval around the sample mean. Besides, what do you mean by $n\approx\lambda$, given they are 88 and 47 respectively? $\endgroup$ – Randel Aug 8 '14 at 19:07
  • 2
    $\begingroup$ Thanks! I have now edited the answer including some specific calculations. The question does not explain how $\lambda$ and n have been obtained, so I made an educated guess. As you say, if n differs too much from $\lambda$ is the first hint that the model may not be Poisson or the measurement was not done right. One way to check it is precisely calculating the 95% confidence interval which, in this case, shows n is outside the interval. $\endgroup$ – jose.angel.jimenez Aug 8 '14 at 20:53
  • 2
    $\begingroup$ I believe the response by jose.angel.jiminez above is incorrect, and arises from misreading the original question. The original poster stated "Observations (n) = 88" -- this was the number of time intervals observed, not the number of events observed overall, or per interval. The average number of events per interval, over the sample of 88 observing intervals, is the lambda given by the original poster. (I'd have included this as a comment to Jose's post, but am too new to the site to be allowed to comment.) $\endgroup$ – user44436 Mar 20 '15 at 18:09
  • $\begingroup$ @user44436 added an answer that was supposed to be a comment. I repost it as a comment so that you can see it and because as a non-answer it may get removed: ------- I believe the response by jose above is incorrect and arises from misreading the original question. The original poster stated Observations (n) = 88 - this was the number of time intervals observed, not the number of events observed overall, or per interval. The average number of events per interval over the sample of 88 observing intervals is the lambda given by the original poster. $\endgroup$ – Mörre May 11 '15 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.