2
$\begingroup$

I've written some code to calculate Poisson confidence limits (a) using Chi-squared, and (b) from first principles, using Poisson's probability mass function equation. However, the two sets of results don't agree. For example, for Lambda=10 and 95% confidence limits, I get:

  95%:
    Chi squared [4.80, 18.39]
    Exact       [3, 17]

The discrepancy is worse for wider confidence limits:

  3 sigma:
    Chi squared [3.08, 23.64]
    Exact       [1, 21]

Note that my 'exact' result shows the first k outside the confidence limits (in other words, the range [4,16] is completely inside the 95% range).

My 95% Chi squared result above agrees with several online Poisson limits calculators. Statpages, for example, also gives [4.80, 18.39].

However, the exact result also appears to be correct. For 95%, the online calculator at Stattrek appears to give the same results as my exact [3,17]. More precisely, here are cumulative probabilities for lambda=10 taken from a table here:

0 - 2 events: negligible
3  1.0%
4  2.9%
...
15 95.1%
16 97.3%
17 98.6%

So, the 95 confidence limits are for 4 to 16 events, inclusive, which agrees with my program output, which says that <= 3 events, or >= 17 events, are outside the 95% limits.

Have I got this wrong somewhere? Is it just that Lambda=10 is too small for the chi-squared method to be exact? If I increase Lambda to 100 I get:

  95%:
    Chi squared [81.36, 121.63]
    Exact       [80, 120]
  3 sigma:
    Chi squared [72.65, 133.83]
    Exact       [70, 131]

It essentially makes no difference. I can live with the fact that the inexact result is continuous, but not with the inaccuracy.

EDIT

Thanks for the comments, everyone. As I understand it, the basic answer is that they both provide confidence limits, but they're different, and I shouldn't expect them to be the same, and should just live with it - correct?

For background, this is for analysing healthcare providers, and finding out if any differ significantly from the average. The important thing here (for me, anyway) is not to point the finger at somebody and say that they're outside the 2 or 3 SD limits, when another analysis could show that they're actually inside the limits. For the same reason, I don't care that a discrete method doesn't give me exact 95% coverage - I just need to positively identify outliers.

My own background isn't stats, but I do understand the exact method, and I'm happy that it gives the "right" answer (notwithstanding the fact that the processes aren't really appropriate for Poisson). However, I don't understand the Poisson/Chi-squared transformation, and I'm not happy with it for this application, because it 'incorrectly' adds outliers at the low end of the range (not to mention missing 'real' outliers at the top). However, it is universally used for exactly this application. Would it be fair for me to say that the exact method is better for this application, and the approximation is simply that, and it is incorrect?

$\endgroup$
  • $\begingroup$ What is your sample size? The $\chi^2$ test is an asymptotic test. $\endgroup$ – Macro May 14 '12 at 12:10
  • 1
    $\begingroup$ Why do you expect them to agree? Different methods for constructing confidence intervals usually give different intervals... $\endgroup$ – MånsT May 14 '12 at 12:42
  • 1
    $\begingroup$ The intervals should be compared in terms of their coverage. How close are they to including the true parameter 95% of the time. It is possible to see a large difference in the interval endpoints without see the true confidence level depart much from the stated level of 95%. Also bear in mind that the Poisson is a discrete distribution on the integers. This means that even the exact methods are not always "exact." $\endgroup$ – Michael Chernick May 14 '12 at 13:43
  • 1
    $\begingroup$ For discrete distributions the exact methods will pick integre end points but it is possible that no such interval has exact 95% coverage. You may have one with 94% coverage and adding the next integer to the interval gives 97%. So the one that gives 97% would be defined as your exact 95% confidence interval. Just like the normal approximation to the binomial approximates a discrete distribution with a continuous one, so too does the chi square do that in approximating the Poisson. $\endgroup$ – Michael Chernick May 14 '12 at 13:45
  • $\begingroup$ In the normal approximation to the binomial a continuity correction is often appled to improve the approximation in small samples. Perhaps a continuity correction could help here too. So to summarize, differences could be due to (1) small sample size and a continuous approximation to a discrete distribution (2) definition of exact for discrete distributions or (3) small difference in coverage producing a large looking difference in the interval and perhaps its width. $\endgroup$ – Michael Chernick May 14 '12 at 13:48
2
$\begingroup$

There is no one "confidence interval"; a confidence interval at level $\alpha$ is any interval with coverage level of $\alpha$ or greater. If you calculate the coverage of the confidence interval $[4, 19]$ (the rounded down / up version of your first interval) for $\lambda=10$, you'll get a coverage probability of approximately 0.967. If you do so with your Poisson-based interval $[3,17]$, you'll get 0.975. They are both 95% confidence intervals; as Michael Chernick has observed (+1), with discrete distributions it is quite possible that no interval has exact coverage. As MansT has observed (+1) , different methods for calculating intervals shouldn't be expected to agree, although they should, in general, provide the targetted coverage.

$\endgroup$
2
$\begingroup$

Why would you use Chi-squared when you can calculate exact values easily? The chi-squared values are based on a central limit approximation. The approximation will get better as $\lambda \to \infty$, but it will never match the exact values.

$\endgroup$
  • $\begingroup$ Because that's what everyone else does :) Seriously. It looks like someone did some Excel macros a long time ago, and everyone else has just copied them. I had assumed that it would take too much computing power to do it directly, but that doesn't seem to be the case. Maybe it was 20 years ago. $\endgroup$ – EML May 14 '12 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.