0
$\begingroup$

I'm trying to fit a Poisson and negative binomial distribution to my data and compare the two; but the problem is that the Poisson fails to capture the overdispersion and the negative binomial seems to overfit and also have difficulty in estimating the dispersion parameter. I was wondering if a weighted combination of the two would yield better results; but in that case how would we estimate the parameters and the weighting factor?

$\endgroup$
  • 1
    $\begingroup$ Hard to see how a distribution with only two parameters could overfit - and, if you try to fit a mixture, you'd be adding a third parameter! Also, the Poisson is a limiting form of the negative binomial, so if the NB doesn't fit well, the Poisson almost certainly won't either. $\endgroup$ – jbowman Dec 23 '13 at 14:43
  • 1
    $\begingroup$ Could you give an example of what you mean by "over-fitting"? What software are you using to estimate the parameters of the negative binomial distribution? $\endgroup$ – probabilityislogic Dec 26 '13 at 13:10
2
$\begingroup$

As your question is related to a Poisson and negative binomial distribution, I assume you are referring to count data, for example a number of goals made during a game, and you also registered the number of training hours of the last year.

Assume you want to model this data using the training hours as explanatory variable, as you assume the player that trains a lot makes more goals. You would assume that this explanatory variable would explain some of the variation in goals made during a game.

However, the Poisson model has as an assumption that the mean of the observed data is equal to the variance. Very often, you end up with a greater variance than demanded by Poisson, and you have to deal with over dispersion (including other predictor variables will further reduce the variance). Negative binomial models can be used in that case.

But even more important is that the zeros in the data have a different "data generating process" than the other counts, as maybe 20% of the kids did not even play during the game and had therefore not even the chance of making a goal. This also leads to lots of unexplained variation, which makes Poisson even less useful. In that case you model the zeros differently, with either zero inflated models or hurdle models.

You can find this summary better explained in this online class: http://www.youtube.com/watch?v=T6cv8n9xBGQ

$\endgroup$
0
$\begingroup$

One suggestion would be to try Weibull, which is more flexible. Although of course it's a continuous distribution, so you may need to take this into account.

Another possibility is to indeed fit a mixture model, possibly to accommodate zero-inflation. If you're using R, try zeroinfl from package pscl.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.