10
$\begingroup$

I was performing a Poisson regression in SAS and found that the Pearson chi-squared value divided by the degrees of freedom was around 5, indicating significant overdispersion. So, I fit a negative binomial model with proc genmod and found the Pearson chi-squared value divided by the degrees of freedom is 0.80. Is this now considered to be underdispersed? If so, how does one go about handling this? I have read a lot about overdispersion and believe I know how to handle this but information on how to handle or determine if there is underdispersion is scant. Can anyone assist?

Thanks.

$\endgroup$
17
+50
$\begingroup$

For a Poisson distribution with mean $\mu$ the variance is also $\mu$. Within the framework of generalized linear models this implies that the variance function is $$V(\mu) = \mu$$ for the Poisson model. This model assumption can be wrong for many different reasons. Overdispersed count data with a variance larger than what the Poisson distribution dictates is, for instance, often encountered.

Deviations from the variance assumption can in a regression context take several forms. The simplest one is that the variance function equals $$V(\mu) = \psi \mu$$ with $\psi > 0$ a dispersion parameter. This is the quasi-Poisson model. It will give the same fitted regression model, but the statistical inference ($p$-values and confidence intervals) is adjusted for over- or underdispersion using an estimated dispersion parameter.

The functional form of the variance function can also be wrong. It could be a second degree polynomial $$V(\mu) = a\mu^2 + b \mu + c,$$ say. Examples include the binomial, the negative binomial and the gamma model. Choosing any of these models as an alternative to the Poisson model will affect the fitted regression model as well as the subsequent statistical inference. For the negative binomial distribution with shape parameter $\lambda > 0$ the variance function is $$V(\mu) = \mu\left( 1 + \frac{\mu}{\lambda}\right).$$ We can see from this that if $\lambda \to \infty$ we get the variance function for the Poisson distribution.

To determine if the variance function for the Poisson model is appropriate for the data, we can estimate the dispersion parameter as the OP suggests and check if it is approximately 1 (perhaps using a formal test). Such a test does not suggest a specific alternative, but it is most clearly understood within the quasi-Poisson model. To test if the functional form of the variance function is appropriate, w e could construct a likelihood ratio test of the Poisson model ($\lambda = \infty$) against the negative binomial model ($\lambda < \infty$). Note that it has a nonstandard distribution under the null hypothesis. Or we could use AIC-based methods in general for comparing non-nested models. Regression-based tests for overdispersion in the Poisson model explores a class of tests for general variance functions.

However, I would recommend to first of all study residual plots, e.g. a plot of the Pearson or deviance residuals (or their squared value) against the fitted values. If the functional form of the variance is wrong, you will see this as a funnel shape (or a trend for the squared residuals) in the residual plot. If the functional form is correct, that is, no funnel or trend, there could still be over- or underdispersion, but this can be accounted for by estimating the dispersion parameter. The benefit of the residual plot is that it suggests more clearly than a test what is wrong with the variance function if anything.

In the OP's concrete case it is not possible to say if 0.8 indicates underdispersion from the given information. Instead of focusing on the 5 and 0.8 estimates, I suggest to first of all investigate the fit of the variance functions of the Poisson model and the negative binomial model. Once the most appropriate functional form of the variance function is determined, a dispersion parameter can be included, if needed, in either model to adjust the statistical inference for any additional over- or underdispersion. How to do that easily in SAS, say, is unfortunately not something I can help with.

$\endgroup$
  • 2
    $\begingroup$ +1, this is good general information. It might be more helpful for the OP if you specifically addressed the OP's explicit questions: (1) is .8 underdispersed; & (2) if so, how to deal w/ that. $\endgroup$ – gung Jan 20 '15 at 22:05
  • $\begingroup$ @gung, I have edited the answer to give more specific advice. You cannot determine if 0.8 is significantly smaller than 1 from the information available, and IMHO focusing on whether the dispersion parameter is 1 is a diversion. My edit explains what I think the OP should focus on instead. $\endgroup$ – NRH Jan 20 '15 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.