0
$\begingroup$

Using MASS::glm.nb (R) I am having problems of convergence with certain dataset, and it seems theta goes very high, resulting in NaNs "In sqrt(1/i)" during the fit. I am trying to understand this and have run some tests to see if NB is a significant better fit than Poisson, and it isn't (p~0.3). My question: is it still appropriate to use a NB when it isn't significantly better than a Poisson, keeping in mind that in this case is better to be conservative? Many thanks!

Also, if anyone knows how to fix those sqrt(1/i) errors that would be great. I can avoid them if I remove the covariates from the model. (I guess the covariates are explaining all of the dispersion?)

$\endgroup$
1
  • $\begingroup$ What exactly are your data and the model? Why did you initially choose negative binomial distribution? $\endgroup$
    – Tim
    Sep 1, 2022 at 17:47

1 Answer 1

0
$\begingroup$

If the conditional (upon all the regressors) variance is less than or equal to the conditional mean, the NB will have a hard or impossible time being fit (which one depends on the algorithm), since there is no parameterization of the NB distribution for which this is the case. The Poisson, however, relies only upon the conditional mean and consequently can be fit regardless of whether the conditional variance equals the conditional mean (the variance-mean relationship of the Poisson distribution.) The error you are seeing is characteristic of this effect. This conclusion is reinforced by the fact that it goes away when you remove the covariates; as you suspect, the covariates appear to be explaining enough of the dispersion to drive the conditional variance below the conditional mean.

Consequently, if you are having problems of this sort fitting the NB distribution, fitting the Poisson is perfectly acceptable. After all, it is the limit of the negative binomial as the variance-mean ratio approaches one!

$\endgroup$
1
  • $\begingroup$ Many thanks for the very clear explanation! $\endgroup$
    – Fede
    Sep 1, 2022 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.