Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 11 down vote favorite
2

Due to the factorial in a poisson distribution, it becomes unpractical to estimate poisson models (for example, using maximum likelihood) when the observations are large. So, for example, if I am trying to estimate a model to explain the number of suicides in a given year (only annual data are available), and say, there are thousands of suicides every year, is it wrong to express suicides in hundreds, so that 2998 would be 29.98 ~= 30? In other words, is it wrong to change the unit of measurement to make the data manageable?

share|improve this question

5 Answers

up vote 9 down vote accepted

When you're dealing with a Poisson distribution with large values of \lambda (its parameter), it is common to use a normal approximation to the Poisson distribution.

As this site mentions, it's all right to use the normal approximation when \lambda gets over 20, and the approximation improves as \lambda gets even higher.

The Poisson distribution is defined only over the state space consisting of the non-negative integers, so rescaling and rounding is going to introduce odd things into your data.

Using the normal approx. for large Poisson statistics is VERY common.

share|improve this answer
up vote 5 down vote

In case of Poisson it is bad, since counts are counts -- their unit is an unity. On the other hand, if you'd use some advanced software like R, its Poisson handling functions will be aware of such large numbers and would use some numerical tricks to handle them.

Obviously I agree that normal approximation is another good approach.

share|improve this answer
up vote 3 down vote

Most statistical packages have a function to calculate the natural logarithm of the factorial directly (e.g. the lfactorial() function in R, the lnfactorial() function in Stata). This allows you to include the constant term in the log-likelihood if you want.

share|improve this answer
In addition, n! = Gamma(n+1) for n >= 0. So try to look for a function called Gamma if you need to calculate the factorial (or log Gamma if you're calculating the log likelihood) – Andre Holzner Aug 30 '10 at 18:43
up vote 2 down vote

I'm afraid you can't do that. As @Baltimark states, with big lambda the distribution will be of more normal shape (symmetric), and with scaling it down it will no longer be poisson distrubution. Try the following code in R:

poi1 = rpois(100000, lambda = 5)  # poisson
poi2 = rpois(100000, lambda = 100)/20 # scaled-down poisson
poi2_dens = density(poi2)

hist(poi1, breaks = 0:30, freq = F, ylim = range(poi2_dens$y))
lines(poi2_dens, col = "red")

The result is below:

enter image description here

You can see that the downscaled poisson (red line) is completely different from the poisson distribution.

share|improve this answer
up vote 1 down vote

You can simply ignore the 'factorial' when using maximum likelihood. Here is the reasoning for your suicides example. Let:

λ : Be the expected number of suicides per year

ki: Be the number of suicides in year i.

Then you would maximize the log-likelihood as:

LL = ∑ ( ki log(λ) - λ - ki! )

Maximizing the above is equivalent to maximizing the following as ki! is a constant :

LL' = ∑ ( ki log(λ) - λ )

Could explain why the factorial is an issue? Am I missing something?

share|improve this answer
You're not missing something if all you're trying to do is estimate the parameter from a set of observations. That was definitely the main idea of the OP's question. However, she was also asking generally (if not rigorously) "how to estimate poisson models". Perhaps she wants to know the value of the pdf at a specific point. In that case, the normal approx. is probably going to be better than scaling the parameter, and the observations by 100, or whatever, if the observations are large enough to make calculating the factorial impractical. – Baltimark Jul 20 '10 at 14:54
@Srikant, you are right, to estimate the parameters the factorial is not an issue, but in general you will want the value of the likelihood for a given model, and you would have to use the factorial for that. Also, for hypothesis testing (e.g. likelihood ratio test) you will need the value of the likelihood. – Vivi Jul 20 '10 at 21:31
@Baltimark: yes, I want to know in general, whether it is valid to change the unit of measurement of Poisson. I was asked this question and I didn't know what to say. – Vivi Jul 20 '10 at 21:33
@Vivi: I am not sure why you would want to compute the likelihood with k_i! included as in most applications (e.g., likelihood ratio test, bayesian estimation) the constant will not matter. In any case, I do not think you can re-scale as you suggested. If I feel otherwise I will update my answer. – user28 Jul 21 '10 at 0:28
@Srikant, I see your point, but some softwares (Eviews, for example) include this by default, and large numbers are an issue you like it or not. I guess I was really after an explanation of why you can or can't do it rather than a way around it, but the discussion has been interesting and instructive nonetheless :) – Vivi Jul 21 '10 at 0:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.