I know of normality tests, but how do I test for "Poisson-ness"?
I have sample of ~1000 non-negative integers, which I suspect are taken from a Poisson distribution, and I would like to test that.
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
First of all my advice is you must refrain from trying out a Poisson distribution just as it is to the data. I suggest you must first make a theory as to why should Poisson distribution fit a particular dataset or a phenomenon.
Once you have established this, the next question is whether the distribution is homogeneous or not. This means whether all parts of the data are handled by the same poisson distribution or is there a variation in this based on some aspect like time or space. Once you have convinced of these aspects, try the following three tests:
search for these and you will find them easily on the net.
Here is a sequence of R commands that may be helpful. Feel free to comment or edit if you spot any mistakes.
set.seed(1) x.poi<-rpois(n=200,lambda=2.5) # a vector of random variables from the Poisson distr. hist(x.poi,main="Poisson distribution") lambda.est <- mean(x.poi) ## estimate of parameter lambda (tab.os<-table(x.poi)) ## table with empirical frequencies freq.os<-vector() for(i in 1: length(tab.os)) freq.os[i]<-tab.os[[i]] ## vector of emprical frequencies freq.ex<-(dpois(0:max(x.poi),lambda=lambda.est)*200) ## vector of fitted (expected) frequencies acc <- mean(abs(freq.os-trunc(freq.ex))) ## absolute goodness of fit index acc acc/mean(freq.os)*100 ## relative (percent) goodness of fit index h <- hist(x.poi ,breaks=length(tab.os)) xhist <- c(min(h$breaks),h$breaks) yhist <- c(0,h$density,0) xfit <- min(x.poi):max(x.poi) yfit <- dpois(xfit,lambda=lambda.est) plot(xhist,yhist,type="s",ylim=c(0,max(yhist,yfit)), main="Poison density and histogram") lines(xfit,yfit, col="red") #Perform the chi-square goodness of fit test #In case of count data we can use goodfit() included in vcd package library(vcd) ## loading vcd package gf <- goodfit(x.poi,type= "poisson",method= "MinChisq") summary(gf) plot(gf,main="Count data vs Poisson distribution")
For a Poisson distribution, the mean equals the variance. If your sample mean is very different from your sample variance, you probably don't have Poisson data. The dispersion test also mentioned here is a formalization of that notion.
If your variance is much larger than your mean, as is commonly the case, you might want to try a negative binomial distribution next.
You can use the dispersion (ratio of variance to the mean) as a test statistic, since the Poisson should give a dispersion of 1. Here is a link to how to use it as a model test.
You can draw a single figure in which the observed and expected frequencies are drawn side by side. If the distributions are very different and you also have a variance-mean ratio bigger than one, then a good candidate is the negative binomial. Read the section Frequency Distributions from
The R Book. It deals with a very similar problem.
I think the main point is the one sidmaestro raises...does the experimental setup or data generation mechanism support the premise that the data might arise from a Poisson distribution.
I'm not a big fan of testing for distributional assumptions, since those tests typically aren't very useful. What seems more useful to me is to make distributional or model assumptions that are flexible and reasonably robust to deviations from the model, typically for purposes of inference. In my experience, it is not that common to see mean=variance, so often the negative binomial model seems more appropriate, and includes the Poisson as a special case.
Another point that is important in going for distributional testing, if that's what you want to do, is to make sure that there aren't strata involved which would make your observed distribution a mixture of other distributions. Individual stratum-specific distributions might appear Poisson, but the observed mixture might not be. An analogous situation from regression only assumes that the conditional distribution of Y|X is normally distributed, and not really the distribution of Y itself.