What method for forecasting error measuring in a poisson process?

Question

There's so much different measures for the forecast error that I kind of lose the sight on which one to use. Out of the following: MAPE, MSE, MASE, MAE, rRMSE, MAD.

I know the MAPE isn't right to use because the value can be 0. My assumption is that the others can all be used for measuring the errors in a Poisson process. I know that the results of MAD and MSE can differ and that the optimal forecasting method for MSE will be the one with the fewest large errors, while MAD will have some of both small and large errors.

Question: Is there a 'best' measurement for a Poisson process and which one should I use? The series have no trend and no seasonality

Answer 1

Don't use the MAD for assessing the accuracy of count data forecasts, especially not for intermittent demands. The expected MAD is minimized by the median of your future distribution, not its mean. If, say you forecast $y\sim\text{Pois}(\lambda)$ with $\lambda<\log 2$, then the median is zero... implying that you will probably get the lowest MAD by a flat zero forecast, which is badly biased and likely useless. Here is a presentation I gave at last year's International Symposium on Forecasting on this issue. Or look at Morlidge (2015, Foresight).

I'd rather use the Mean Squared Error (MSE) or its root (RMSE) or perhaps a scaled variant. These are at least minimized by the expectation of the future distribution.

Final piece of shameless self-promotion: I have an article in the IJF (Kolassa, 2016) which looks at forecasting low volume count data (mostly intermittent), at different accuracy measures and different forecasting methods, including various flavors of Poisson models. This may be useful to you.