Skip to main content
Cross Validated

Return to Answer

Bounty Ended with 50 reputation awarded by CommunityBot
Edited in light of gung's comment
Source Link
NRH
NRH
  • 18.6k
  • 73
  • 81

For a Poisson distribution with mean $\mu$ the variance is also $\mu$. Within the framework of generalized linear models this implies that the variance function is $$V(\mu) = \mu$$ for the Poisson model. This model assumption can be wrong for many different reasons. Overdispersed count data with a variance larger than what the Poisson distribution dictates is, for instance, often encountered.

Deviations from the variance assumption can in a regression context take several forms. The simplest one is that the variance function equals $$V(\mu) = \psi \mu$$ with $\psi > 0$ a dispersion parameter. This is the quasi-Poisson model. It will give the same fitted regression model, but the statistical inference ($p$-values and confidence intervals) is adjusted for over- or underdispersion using an estimated dispersion parameter.

The functional form of the variance function can also be wrong. It could be a second degree polynomial $$V(\mu) = a\mu^2 + b \mu + c,$$ say. Examples include the binomial, the negative binomial and the gamma model. Choosing any of these models as an alternative to the Poisson model will affect the fitted regression model as well as the subsequent statistical inference. For the negative binomial distribution with shape parameter $\lambda > 0$ the variance function is $$V(\mu) = \mu\left( 1 + \frac{\mu}{\lambda}\right).$$ We can see from this that if $\lambda \to \infty$ we get the variance function for the Poisson distribution.

To determine if the variance function for the Poisson model is appropriate for the data, we can estimate the dispersion parameter as the OP suggests and check if it is approximately 1 (perhaps using a formal test). Such a test does not suggest a specific alternative, but it is most clearly understood within the quasi-Poisson model. WeTo test if the functional form of the variance function is appropriate, w e could construct a likelihood ratio test of the Poisson model ($\lambda = \infty$) against the negative binomial model ($\lambda < \infty$). Note that it has a nonstandard distribution under the null hypothesis. Or we could use AIC-based methods in general for comparing non-nested models. Regression-based tests for overdispersion in the Poisson model explores a class of tests for general variance functions.

However, I would recommend to first of all study residual plots, e.g. a plot of the Pearson or deviance residuals (or their squared value) against the fitted values. If the functional form of the variance is wrong, you will see this as a funnel shape (or a trend for the squared residuals) in the residual plot. If the functional form is correct, that is, no funnel or trend, there could still be over- or underdispersion, but this can be accounted for by estimating the dispersion parameter. The clear benefit of the residual plot is that it suggests more clearly than a test what is wrong with the variance function if anything.

In the OP's concrete case it is not possible to say if 0.8 indicates underdispersion from the given information. Instead of focusing on the 5 and 0.8 estimates, I suggest to first of all investigate the fit of the variance functions of the Poisson model and the negative binomial model. Once the most appropriate functional form of the variance function is determined, a dispersion parameter can be included, if needed, in either model to adjust the statistical inference for any additional over- or underdispersion. How to do that easily in SAS, say, is unfortunately not something I can help with.

For a Poisson distribution with mean $\mu$ the variance is also $\mu$. Within the framework of generalized linear models this implies that the variance function is $$V(\mu) = \mu$$ for the Poisson model. This model assumption can be wrong for many different reasons. Overdispersed count data with a variance larger than what the Poisson distribution dictates is, for instance, often encountered.

Deviations from the variance assumption can in a regression context take several forms. The simplest one is that the variance function equals $$V(\mu) = \psi \mu$$ with $\psi > 0$ a dispersion parameter. This is the quasi-Poisson model. It will give the same fitted regression model, but the statistical inference ($p$-values and confidence intervals) is adjusted for over- or underdispersion using an estimated dispersion parameter.

The functional form of the variance function can also be wrong. It could be a second degree polynomial $$V(\mu) = a\mu^2 + b \mu + c,$$ say. Examples include the binomial, the negative binomial and the gamma model. Choosing any of these models as an alternative to the Poisson model will affect the fitted regression model as well as the subsequent statistical inference. For the negative binomial distribution with shape parameter $\lambda > 0$ the variance function is $$V(\mu) = \mu\left( 1 + \frac{\mu}{\lambda}\right).$$ We can see from this that if $\lambda \to \infty$ we get the variance function for the Poisson distribution.

To determine if the variance function for the Poisson model is appropriate for the data, we can estimate the dispersion parameter as the OP suggests and check if it is approximately 1 (perhaps using a formal test). Such a test does not suggest a specific alternative, but it is most clearly understood within the quasi-Poisson model. We could construct a likelihood ratio test of the Poisson model ($\lambda = \infty$) against the negative binomial model ($\lambda < \infty$). Note that it has a nonstandard distribution under the null hypothesis. Or we could use AIC-based methods in general for comparing non-nested models. Regression-based tests for overdispersion in the Poisson model explores a class of tests for general variance functions.

However, I would recommend to first of all study residual plots, e.g. a plot of the Pearson or deviance residuals (or their squared value) against the fitted values. If the functional form of the variance is wrong, you will see this as a funnel shape (or a trend for the squared residuals) in the residual plot. If the functional form is correct, that is, no funnel or trend, there could still be over- or underdispersion, but this can be accounted for by estimating the dispersion parameter. The clear benefit of the residual plot is that it suggests more clearly than a test what is wrong with the variance function if anything.

For a Poisson distribution with mean $\mu$ the variance is also $\mu$. Within the framework of generalized linear models this implies that the variance function is $$V(\mu) = \mu$$ for the Poisson model. This model assumption can be wrong for many different reasons. Overdispersed count data with a variance larger than what the Poisson distribution dictates is, for instance, often encountered.

Deviations from the variance assumption can in a regression context take several forms. The simplest one is that the variance function equals $$V(\mu) = \psi \mu$$ with $\psi > 0$ a dispersion parameter. This is the quasi-Poisson model. It will give the same fitted regression model, but the statistical inference ($p$-values and confidence intervals) is adjusted for over- or underdispersion using an estimated dispersion parameter.

The functional form of the variance function can also be wrong. It could be a second degree polynomial $$V(\mu) = a\mu^2 + b \mu + c,$$ say. Examples include the binomial, the negative binomial and the gamma model. Choosing any of these models as an alternative to the Poisson model will affect the fitted regression model as well as the subsequent statistical inference. For the negative binomial distribution with shape parameter $\lambda > 0$ the variance function is $$V(\mu) = \mu\left( 1 + \frac{\mu}{\lambda}\right).$$ We can see from this that if $\lambda \to \infty$ we get the variance function for the Poisson distribution.

To determine if the variance function for the Poisson model is appropriate for the data, we can estimate the dispersion parameter as the OP suggests and check if it is approximately 1 (perhaps using a formal test). Such a test does not suggest a specific alternative, but it is most clearly understood within the quasi-Poisson model. To test if the functional form of the variance function is appropriate, w e could construct a likelihood ratio test of the Poisson model ($\lambda = \infty$) against the negative binomial model ($\lambda < \infty$). Note that it has a nonstandard distribution under the null hypothesis. Or we could use AIC-based methods in general for comparing non-nested models. Regression-based tests for overdispersion in the Poisson model explores a class of tests for general variance functions.

However, I would recommend to first of all study residual plots, e.g. a plot of the Pearson or deviance residuals (or their squared value) against the fitted values. If the functional form of the variance is wrong, you will see this as a funnel shape (or a trend for the squared residuals) in the residual plot. If the functional form is correct, that is, no funnel or trend, there could still be over- or underdispersion, but this can be accounted for by estimating the dispersion parameter. The benefit of the residual plot is that it suggests more clearly than a test what is wrong with the variance function if anything.

In the OP's concrete case it is not possible to say if 0.8 indicates underdispersion from the given information. Instead of focusing on the 5 and 0.8 estimates, I suggest to first of all investigate the fit of the variance functions of the Poisson model and the negative binomial model. Once the most appropriate functional form of the variance function is determined, a dispersion parameter can be included, if needed, in either model to adjust the statistical inference for any additional over- or underdispersion. How to do that easily in SAS, say, is unfortunately not something I can help with.

Source Link
NRH
NRH
  • 18.6k
  • 73
  • 81

For a Poisson distribution with mean $\mu$ the variance is also $\mu$. Within the framework of generalized linear models this implies that the variance function is $$V(\mu) = \mu$$ for the Poisson model. This model assumption can be wrong for many different reasons. Overdispersed count data with a variance larger than what the Poisson distribution dictates is, for instance, often encountered.

Deviations from the variance assumption can in a regression context take several forms. The simplest one is that the variance function equals $$V(\mu) = \psi \mu$$ with $\psi > 0$ a dispersion parameter. This is the quasi-Poisson model. It will give the same fitted regression model, but the statistical inference ($p$-values and confidence intervals) is adjusted for over- or underdispersion using an estimated dispersion parameter.

The functional form of the variance function can also be wrong. It could be a second degree polynomial $$V(\mu) = a\mu^2 + b \mu + c,$$ say. Examples include the binomial, the negative binomial and the gamma model. Choosing any of these models as an alternative to the Poisson model will affect the fitted regression model as well as the subsequent statistical inference. For the negative binomial distribution with shape parameter $\lambda > 0$ the variance function is $$V(\mu) = \mu\left( 1 + \frac{\mu}{\lambda}\right).$$ We can see from this that if $\lambda \to \infty$ we get the variance function for the Poisson distribution.

To determine if the variance function for the Poisson model is appropriate for the data, we can estimate the dispersion parameter as the OP suggests and check if it is approximately 1 (perhaps using a formal test). Such a test does not suggest a specific alternative, but it is most clearly understood within the quasi-Poisson model. We could construct a likelihood ratio test of the Poisson model ($\lambda = \infty$) against the negative binomial model ($\lambda < \infty$). Note that it has a nonstandard distribution under the null hypothesis. Or we could use AIC-based methods in general for comparing non-nested models. Regression-based tests for overdispersion in the Poisson model explores a class of tests for general variance functions.

However, I would recommend to first of all study residual plots, e.g. a plot of the Pearson or deviance residuals (or their squared value) against the fitted values. If the functional form of the variance is wrong, you will see this as a funnel shape (or a trend for the squared residuals) in the residual plot. If the functional form is correct, that is, no funnel or trend, there could still be over- or underdispersion, but this can be accounted for by estimating the dispersion parameter. The clear benefit of the residual plot is that it suggests more clearly than a test what is wrong with the variance function if anything.