Is Mean Square Prediction Error acceptable to use if predicted values are continuous but actual observed values are discrete?

Question

I would like to compare the predictive power of 2 models. The models are meant to model count data, so the actual observed values are discrete. However both models are designed such that they output predicted values that are continuous (i.e. real numbers rather than non negative integers, as they are based on some expected value probability type computation without rounding). I have a dataset with actual observed values, and the predicted values given by both models. I have no further information about the models (i.e. the number of predictors or the specific type of predictive models used).

Is a standard Mean Squared Error or Root Mean Square Error acceptable to use here? Or is there a better way to compare the 2 models?

Answer 1

Absolutely yes.

The (R)MSE elicits (=rewards) unbiased expectation forecasts. Expected sales could definitely be fractional, even if observed sales are integer.

The simplest example would be if sales were, say, Poisson distributed with a mean of 0.2. Then the point forecast that minimizes the expected (R)MSE will be 0.2 - exactly as you want it. Other error measures will pull you towards other forecasts, e.g., the MAE will be minimized in expectation by a flat zero forecast.

There is more on how to think about error measures in Kolassa (2020, IJF).