11
$\begingroup$

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my case. I selected the model with the best AIC. Finally, I was interested how a linear model (GLM with Gaussian distribution and identity link) would perform in comparisson to the count models. The linear model obviously does not fulfil the distributional assumptions as well as the count models. However, now I find model AICs of arround -150 suggesting much better fit.

Should I use the linear model based on AIC despite the fact that basic distributional assumptions are not met? Are the AICs comparable?

I used a GLMM function for fitting: http://glmmadmb.r-forge.r-project.org/ (my data are clustered).

$\endgroup$

1 Answer 1

12
$\begingroup$

You cannot use likelihood-based statistics like AIC to compare across models with different likelihood functions - the underlying formulas are different. In linear regression, the likelihood function is the normal density function, in Poisson regression it is the Poisson function. That will account for the differences in the AIC probably more than any differences in fit.

Before you decide to even use a linear model, you need to make sure that the residuals from the model are normally distributed (you can proxy that by looking at the distribution of the outcome variable, though keep in mind it isn't the same). If they are not normally distributed, or close enough for the eye, then you can't use a normal regression model to do any hypothesis testing.

Assuming that it is approximately normal, I would take a two broad approaches to choose the model to report.

1) Predicted outcomes. Estimate the predicted outcomes of each model and compare. Does the linear model have better predictive ability? You may want to do this in a cross-validation framework, where you "train" your model on part of your data and use the rest for prediction.

2) Intuitive interpretation of coefficients. Poisson coefficients can be complicated to understand - they are not the change in number of y but rather a proportional change. Depending on your context this may be more or less useful. Sometimes it is worth sacrificing fit if your model can be more easily interpreted by the end-user - for example, some researchers are willing to avoid the complexity of logit and probit models for the easier-to-interpret coefficients in a linear probability model, even though the LPM has tons of setbacks. Think about who your audience is, what is your context, what is your research question, etc., as you make these decisions.

EDIT: I forgot to add this paper, which gives a good comparison across a range of different count models and may be helpful.

$\endgroup$
6
  • $\begingroup$ @robin.datadrivers - does the same logic mean that comparing AIC between poisson and negative binomial models would be meaningless? the likelihood functions are similar but slightly different... $\endgroup$ Commented Aug 21, 2015 at 7:57
  • 5
    $\begingroup$ I am not sure I agree with your first sentence. I must admit I do not know all the assumptions behind the AIC but surely the models do not have to be in the same class of models to be comparable. The first thing to look at is that the response variable is exactly the same (not a transformation, not another sample), but other than that it is not so trivial. Here is a related discussion, unfortunately without a very detailed answer. $\endgroup$ Commented Oct 19, 2015 at 18:09
  • $\begingroup$ @RichardHardy: See Likelihood comparable across different distributions. $\endgroup$
    – Scortchi
    Commented Jan 9, 2019 at 15:43
  • 1
    $\begingroup$ @Scortchi, The discussion of measures is a little too heavy for me. But here are my takeaways, let us see if they make sense. (1) Calculate the likelihood exactly (do not drop constants) - obvious. (2) Use exactly the same data - obvious to me. (3) Do not mix discrete vs. continuous models - tough. I think (3) is related to (2) in a sense that the likelihood for a continuous model is the likelihood not only for the data points at hand but also for all the unobserved ones in between. Does that make sense? And what is your take on my criticism of the answer here, do you support it? $\endgroup$ Commented Jan 9, 2019 at 17:53
  • $\begingroup$ @RichardHardy: I'm not well versed in measure theory but I find 3 obvious: you can't compare a mass with a density. I do agree with your criticism - it's not clear from "different likelihood functions" what kind of difference stops you comparing them. $\endgroup$
    – Scortchi
    Commented Jan 9, 2019 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.