My response variable $y_i$ denotes the number of articles produced by journalist $i$ in the last two years. It is a count variable with fixed exposure hence why I chose to use Poisson regression.

I have a dataset containing the number of articles produced by numerous different journalists and I also have values of other variables (my predictor variables).

I have calculated the sample mean and sample variance of the variable $y$, and the sample variance is nearly double (over-dispersion). Is this a good enough reason for me to use negative Binomial regression? Should I consider anything else? The deviance of the model also drops by a noticeable amount when I use NB over Poisson.

This is my first time considering using a negative Binomial regression so I apologise if the question is very basic.

  • 1
    $\begingroup$ Welcome to our site. The sample variance is large because it incorporates variation among the predictor variables, too: thus, it is inappropriate for selecting the model. Consider applying what you initially thought was a good model and conducting diagnostic analyses on its residuals: that is what matters. $\endgroup$ – whuber Nov 28 '18 at 22:30
  • $\begingroup$ @whuber Thank you for your response but I am not convinced by that explanation. I am only looking at the sample variance and mean of the number of articles produced; I do not consider the predictors when I do this. $\endgroup$ – Abdul Miah Nov 28 '18 at 22:34
  • 1
    $\begingroup$ To expound on @whuber's point, over-dispersion in regression analysis refers to the distribution of the response within predictors- whereas it sounds like you've considered the distribution of the response across predictors. $\endgroup$ – khol Nov 28 '18 at 22:35
  • $\begingroup$ @khol Interesting. How could I correctly test for over-dispersion, given a glm? $\endgroup$ – Abdul Miah Nov 28 '18 at 22:38

Formally, i don't think you can go off your raw sample variance and mean, because the distribution is conditional i.e. $Y|X \thicksim P(\lambda)$

I can be totally wrong on the first point, so someone correct me. But generally as a rule of thumb test for overdispersion is to fit your model and divide the residual deviance by the residual degrees of freedom; if it's much larger than 1 then it's probably overdispersed.

A more formal hypothesis test is to simply test if the negative binomial fits better. That is, the Poisson is the Negative Binomial when $k \rightarrow 0$. So, we are actually testing if

$H_0: k = 0$ and $H_1: k > 0$

This can be tested using the log likelihood ratio test:

$LR=2(\ell_{NB}-\ell_p)$ which is distributed $\chi_p^2$ where $p$ is the number of parameters. Only tricky thing is the cutoff is at $\alpha=0.1$ instead of $0.05$ because the distribution of LR has a mass of $0.5$ at $k=0$ and a half-$\chi_p^2$ distribution above zero. Basically just fit two models and use the log likelihood reatio test to compare their deviances.

In R, you can either compute it by hand using the logLik to pull out the loglikelihoods and substituting into the formmula and then looking up the chi squared criitical value at 0.1. Or, the pscl package has a odTest command to test this for you. See more here, but I haven't tried that one myself.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.