For context, I need some metrics that can compare a standard Poisson regression (with population offset) to a random forest regressor with Poisson criterion.

The test predictions for both methods are output in terms of rates (i.e., counts per population) which is, for all intents and purposes, what I am actually interested in.

However, my understanding is that metrics such as deviance are calculated in terms of raw counts. Is it valid to instead use the rates here?

To illustrate this a bit further, if I had the following observation:

Would evaluation measures be in terms of the error 110 - 100 = 10 ?

Or instead (110/2000) - (100/2000) = 0.005 ?

Furthermore, since these rates are continuous, could mean square error (or total error) be used as a valid performance measure as well?

Thanks so much!