41
$\begingroup$

I know of normality tests, but how do I test for "Poisson-ness"?

I have sample of ~1000 non-negative integers, which I suspect are taken from a Poisson distribution, and I would like to test that.

$\endgroup$
18
$\begingroup$

First of all my advice is you must refrain from trying out a Poisson distribution just as it is to the data. I suggest you must first make a theory as to why should Poisson distribution fit a particular dataset or a phenomenon.

Once you have established this, the next question is whether the distribution is homogeneous or not. This means whether all parts of the data are handled by the same poisson distribution or is there a variation in this based on some aspect like time or space. Once you have convinced of these aspects, try the following three tests:

  1. likelihood ratio test using a chi square variable
  2. use of conditional chi-square statistic; also called poisson dispersion test or variance test
  3. use of the neyman-scott statistic, that is based on a variance stabilizing transformation of the poisson variable

search for these and you will find them easily on the net.

$\endgroup$
  • 3
    $\begingroup$ "whether the distribution is homogeneous or not... Once you have convinced of these aspects" Could you comment a little how to determine this? $\endgroup$ – means-to-meaning Jan 16 '15 at 18:29
12
$\begingroup$

Here is a sequence of R commands that may be helpful. Feel free to comment or edit if you spot any mistakes.

set.seed(1)
x.poi<-rpois(n=200,lambda=2.5) # a vector of random variables from the Poisson distr.

hist(x.poi,main="Poisson distribution")

lambda.est <- mean(x.poi) ## estimate of parameter lambda
(tab.os<-table(x.poi)) ## table with empirical frequencies


freq.os<-vector()
for(i in 1: length(tab.os)) freq.os[i]<-tab.os[[i]]  ## vector of emprical frequencies

freq.ex<-(dpois(0:max(x.poi),lambda=lambda.est)*200) ## vector of fitted (expected) frequencies

acc <- mean(abs(freq.os-trunc(freq.ex))) ## absolute goodness of fit index acc
acc/mean(freq.os)*100 ## relative (percent) goodness of fit index

h <- hist(x.poi ,breaks=length(tab.os))
xhist <- c(min(h$breaks),h$breaks)
yhist <- c(0,h$density,0)
xfit <- min(x.poi):max(x.poi)
yfit <- dpois(xfit,lambda=lambda.est)
plot(xhist,yhist,type="s",ylim=c(0,max(yhist,yfit)), main="Poison density and histogram")
lines(xfit,yfit, col="red")

#Perform the chi-square goodness of fit test 
#In case of count data we can use goodfit() included in vcd package
library(vcd) ## loading vcd package
gf <- goodfit(x.poi,type= "poisson",method= "MinChisq")
summary(gf)
plot(gf,main="Count data vs Poisson distribution")
$\endgroup$
  • 2
    $\begingroup$ This line: freq.ex<-(dpois(0:max(x.poi),lambda=lambda.est)*200) produces errors with some real world data, because the length of freq.ex won't match freq.obs on this line acc <- mean(abs(freq.os-trunc(freq.ex))). I adapted this line to freq.ex<-(dpois(seq(0,max(x.poi))[seq(0:max(x.poi)) %in% x.poi],lambda=lambda.est)*200) but something still isn't quite right because goodfit produces warnings. $\endgroup$ – russellpierce Aug 16 '10 at 17:15
  • $\begingroup$ Using your code I receive a goodess of fit index of 121. But it's relative, isn't? So it should be between 0 an 100, right? $\endgroup$ – MERose Sep 3 '15 at 22:13
6
$\begingroup$

I suppose the easiest way is just to do a chi-squared Goodness of fit test.

In fact here's nice java applet that will do just that!

$\endgroup$
5
$\begingroup$

You can use the dispersion (ratio of variance to the mean) as a test statistic, since the Poisson should give a dispersion of 1. Here is a link to how to use it as a model test.

$\endgroup$
  • $\begingroup$ +1 thank you. often I get some "weired" results, for example, a normal distribution gets a higher p-value then a poisson one, where lambda is relatively small (so by looks only the normal and poisson are not similiar at all) $\endgroup$ – David B Aug 4 '10 at 6:51
  • 6
    $\begingroup$ 404 NOT FOUND!! $\endgroup$ – Rodrigo Mar 23 '16 at 20:02
5
$\begingroup$

For a Poisson distribution, the mean equals the variance. If your sample mean is very different from your sample variance, you probably don't have Poisson data. The dispersion test also mentioned here is a formalization of that notion.

If your variance is much larger than your mean, as is commonly the case, you might want to try a negative binomial distribution next.

$\endgroup$
  • 5
    $\begingroup$ And so if the mean is the same as the variance, could you conclude that the data was Poisson? Hardly! $\endgroup$ – PeterR Aug 3 '10 at 14:59
  • $\begingroup$ True. Necessary but not sufficient. $\endgroup$ – John D. Cook Aug 3 '10 at 16:44
2
$\begingroup$

You can draw a single figure in which the observed and expected frequencies are drawn side by side. If the distributions are very different and you also have a variance-mean ratio bigger than one, then a good candidate is the negative binomial. Read the section Frequency Distributions from The R Book. It deals with a very similar problem.

$\endgroup$
1
$\begingroup$

I think the main point is the one sidmaestro raises...does the experimental setup or data generation mechanism support the premise that the data might arise from a Poisson distribution.

I'm not a big fan of testing for distributional assumptions, since those tests typically aren't very useful. What seems more useful to me is to make distributional or model assumptions that are flexible and reasonably robust to deviations from the model, typically for purposes of inference. In my experience, it is not that common to see mean=variance, so often the negative binomial model seems more appropriate, and includes the Poisson as a special case.

Another point that is important in going for distributional testing, if that's what you want to do, is to make sure that there aren't strata involved which would make your observed distribution a mixture of other distributions. Individual stratum-specific distributions might appear Poisson, but the observed mixture might not be. An analogous situation from regression only assumes that the conditional distribution of Y|X is normally distributed, and not really the distribution of Y itself.

$\endgroup$
  • $\begingroup$ Your last point about regression would only be true if X is random. If X is fixed then Y would also be a normal. no? $\endgroup$ – user28 Aug 4 '10 at 15:28
  • $\begingroup$ Yes, this is true, but for general regression problems (as opposed to anova or designed problems), X really isn't fixed but are observations from the underlying process. However, for the Poisson case, the point still holds, since mixtures of Poissons aren't necessarily Poisson. $\endgroup$ – Abhijit Aug 4 '10 at 22:23
0
$\begingroup$

Yet another way to test this is with a quantile quantile plot. In R, there is qqplot. This directly plots your values against a normal distribution with similar mean and sd

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.