How can one say if a model is poor based on RMSE value

I have a general question about the value of using RMSE to see if a forecasting model is poor. I used the forecast package in R to find forecasting models for different datasets and used RMSE for test data to see how far the predictions are from their actual values. Some data sets have values of around 500 for daily number of orders and RMSE for them is around 50s. But I have some datasets that have an average of 100 orders per day and the RMSE is around 10.

How can I say if these numbers for RMSE are high? I just want to make sure that the fitted model works fine and I thought maybe RMSE is the best factor to see that.

I should note that I don't want to compare different models. I used the tbats package in R which gave me the best fitted model.

1 Answer

There is no way you can decide whether a given RMSE (or other forecast accuracy measure) is high or low as such. You will need to compare this RMSE to the forecast accuracy that is reasonably achievable for a given time series. This has been called "forecastability", and Sean Schubert has written a series of articles on the concept for Foresight, the practitioner-oriented publication of the International Institute of Forecasters. I'd recommend that you look at these papers (full disclosure: I am an Associate Editor of Foresight).

• Thank you Stephan. I will look at the references you mentioned. just a quick question. Is it usefull if I get an average over residuals of the predictions for test data and say like the error ratio is about 30% in average and based on the decision maker's goal, we can decide if the model works fine or not. Right now, for the industry I want to use forecasting, they are almost 80% wrong in predictions and even if a forecasting model provides 50% of error, still its doing better than their no forecasting system as it reduces the wrong predictions by 30%. – user12 Sep 24 '14 at 13:47
• @Fateme: that sounds reasonable. However, I'd advise you to be very cautious about trusting industry benchmarks for forecasting accuracy. – Stephan Kolassa Sep 24 '14 at 14:13
• I want to fit a model to this industry data so I need to use this data to fit a model to them. Honestly I dont undrestand why you said that I shouldnt use industry benchmarks for forecasting accuracy. – user12 Sep 24 '14 at 14:45
• When you wrote that "they are almost 80% wrong in predictions", it sounded like you were using published benchmarks on forecasting accuracy for an industry. If I understand your last comment correctly, you already have "standard" forecasts for the data you are using your models on, correct? Then you can disregard my last comment. And you could directly compare your and the standard forecasts and see how often yours is better. – Stephan Kolassa Sep 24 '14 at 14:48
• I dont have a standard forecast. It was an oral statement mentioned by an industry practitioner. They believe that they are 80% wrong in predictions for their next day number of orders. – user12 Sep 24 '14 at 15:06