Negative binomial or Poisson

Question

Using MASS::glm.nb (R) I am having problems of convergence with certain dataset, and it seems theta goes very high, resulting in NaNs "In sqrt(1/i)" during the fit. I am trying to understand this and have run some tests to see if NB is a significant better fit than Poisson, and it isn't (p~0.3). My question: is it still appropriate to use a NB when it isn't significantly better than a Poisson, keeping in mind that in this case is better to be conservative? Many thanks!

Also, if anyone knows how to fix those sqrt(1/i) errors that would be great. I can avoid them if I remove the covariates from the model. (I guess the covariates are explaining all of the dispersion?)

Answer 1

If the conditional (upon all the regressors) variance is less than or equal to the conditional mean, the NB will have a hard or impossible time being fit (which one depends on the algorithm), since there is no parameterization of the NB distribution for which this is the case. The Poisson, however, relies only upon the conditional mean and consequently can be fit regardless of whether the conditional variance equals the conditional mean (the variance-mean relationship of the Poisson distribution.) The error you are seeing is characteristic of this effect. This conclusion is reinforced by the fact that it goes away when you remove the covariates; as you suspect, the covariates appear to be explaining enough of the dispersion to drive the conditional variance below the conditional mean.

Consequently, if you are having problems of this sort fitting the NB distribution, fitting the Poisson is perfectly acceptable. After all, it is the limit of the negative binomial as the variance-mean ratio approaches one!