What measurement of error should I use to compare predicted model results to actual measurements (and why)?

Question

Lets say I have a time-series dataset of measurements g that varies with time t and I also have a time-series dataset of predictions of these measurements (lets call this g1) that again varies with time t.

I want to be able to quantitatively say how well the predicted data g1 matches the measured data g using a statistic that estimates error (for example $MSE$ or $R^2$). However as someone with only a brief introductory background in statistics I do not understand the differences in what different error statistics are telling me.

So what would be the best error measurement to find this (with an explanation of what that statistic is actually telling me)?

Answer 1

This is a rather broad question.

Generally speaking, there are a number of point forecast accuracy measures, all of which measure different things. It helps to take a step back and realize that you (or your forecasting algorithm) is in fact working with an entire predictive density for each future time point, which it then summarized with a single number, the point forecast. Different error measures are minimized by different one-number summaries of that distribution.

• The MSE is minimized by the expectation of the predictive density. So use the MSE if you want an unbiased forecast.
• The Mean Absolute Error (MAE) is minimized by the median of the predictive density. Use the MAE if you need a point forecast at the median of the future distribution.
• The Mean Absolute Percentage Error (MAPE) is minimized by a rather exotic functional: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? This thread also contains a visualization of the effects I am discussing.
• $R^2$ measure the percentage of variation in your data captured by your forecasts. I don't think it is very informative as an error measure.

Here is more information on accuracy measurement.

I personally would recommend that you go with the MSE, which will draw you towards unbiased forecasts. If you need to report other KPIs, which can look easier to interpret (which I would argue is a false impression), then at least also assess whether your forecasts are systematically biased.