How do we compare count models for prediction and inference?

Question

I have estimated a number of count models on a data, including Poisson, Zero-Inflated Poisson (ZIP), mixed-effects Poisson, mixed-effects ZIP and, a few different versions of each of these based on inclusion of different subsets of explanatory variables. The objectives of the study are inference and prediction. I have a future-year data for prediction comparison. I have used AIC and Mean Squared Error (MSE) as comparison tools. My questions are: 1) Is AIC an appropriate approach for comparing different types of models on a data (Poisson vs. ZIP) and mixed-effects vs. only-fixed-effects models? 2) Is MSE a good measure for prediction comparison (my problem with MSE is that it is unit is not tangible, meaning I don't know if, for example, MSE = 0.0031 and MSE = 0.0035 are actually significantly different)?

Answer 1

I am not completely sure about comparing AICs, especially between mixed and non-mixed models. I suspect that this is invalid. If at all, use the exact same software package to fit all models, by constraining some as necessary, since different packages may calculate information criteria in different ways. (Good luck finding something that can handle both zero inflation and mixed models.)

I'd rather compare models by assessing predictive performance on holdout samples, if you have enough data.

MSE is a valid measure to compare point predictions, if you are mainly interested in the expectation forecast. If you want to assess whether a small difference is significant, you can bootstrap either just the predictions, by resampling the holdout set for a fixed fitted model, or the model fit, by resampling the training set, fitting a model and assessing it on the fixed holdout set. Or even do both. In each case, you will get an entire distribution of MSEs and can check how much they overlap.

However, different models may give very similar point expectation predictions (and therefore MSEs) but have very different notions of variability. For instance, a plain vanilla Poisson distribution and a ZIP with a different Poisson parameter may well have the same expectation, but very different histograms. I'd recommend you assess the entire predictive distribution using scoring-rules. I have written a little paper on evaluating discrete predictive distributions (Kolassa, 2016, International Journal of Forecasting) that may be helpful.

Answer 2

Here's what I would do. I'd look at the results of each model against each other and against the actual data graphically and see what was going on. I'd rely on my intuition and substantive knowledge to see if the more complex models were really better in a meaningful way than the less complex ones.

Graphs could include Tukey Mean Difference plots, quantile-quantile plots, and various other that would depend on the exact situation.