In Poisson models with an offset, should performance metrics (such as deviance) be calculated in terms of raw counts or counts per exposure?

Question

For context, I need some metrics that can compare a standard Poisson regression (with population offset) to a random forest regressor with Poisson criterion.

The test predictions for both methods are output in terms of rates (i.e., counts per population) which is, for all intents and purposes, what I am actually interested in.

However, my understanding is that metrics such as deviance are calculated in terms of raw counts. Is it valid to instead use the rates here?

To illustrate this a bit further, if I had the following observation:

x	exposure	count predicted	count actual
42	2000	100	110

Would evaluation measures be in terms of the error 110 - 100 = 10 ?

Or instead (110/2000) - (100/2000) = 0.005 ?

Furthermore, since these rates are continuous, could mean square error (or total error) be used as a valid performance measure as well?

Answer 1

Your intuition is good! It is valid to use the rates here given that both methods use them as the same response variable. Ultimately a metric value on its own is useless unless it can be compared to some known reference/baseline point.

Yes, you are correct that using (R)MSE is fine. MAE is often reported in count regression as it is more interpretable to some people. Notice though that MAE is not a proper scoring rule (see Predictive Model Assessment for Count Data by Czado et al. (2009) for more details).

A final point to make is that maybe looking at some visual diagnostics can be very helpful. The two I would suggest would be:

1. A calibration plot of our predictions; we want to avoid models that tend to under/over-estimate parts of our data. Usually, we use such plots for risk estimation models but they are equally applicable for count models too (see Predicting good probabilities with supervised learning by Niculescu-Mizil & Caruana (2006) for more details - it concerns probabilities but generalises to counts/rates almost directly.).
2. A rootgram, allowing us to see zero-inflation and over/under-dispersion more easily - sometimes referred as "hanging rootogram" (see Visualizing Count Data Regressions Using Rootograms by Kleiber & Zeileis (2016) for more details).