# Using r, which metric should I use for determining the best model?

library("forecast")
ss<-c(29,36,36,48,93,28,35,28,37,50,37,3,25,28,40,45,38,43,34,44,43,25,33,34)
ss<-ts(ss,f=12,start=c(2016,1))
for (i in 1:12){
ssfc <- ses(ss[c(i:(11+i))],h=1)
ssfc2 <- meanf(ss[c(i:(11+i))],h=1)
ssfc3 <- naive(ss[c(i:(11+i))],h=1)
ssfc4 <- snaive(ss[c(i:(11+i))],h=1)
ssfc5 <- rwf(ss[c(i:(11+i))],h=1,drift=TRUE)
ssfc6 <- croston(ss[c(i:(11+i))],h=1)
ssfc8 <- holt(ss[c(i:(11+i))],h=1)
ssfc9<-holt(ss[c(i:(11+i))],h=1,damped=TRUE)
print(round(accuracy(ssfc),4))
print(ssfc8[["model"]])
print(round(accuracy(ssfc2),4))
print(round(accuracy(ssfc3),4))
print(round(accuracy(ssfc4),4))
print(round(accuracy(ssfc5),4))
print(round(accuracy(ssfc6),4))
print(round(accuracy(ssfc8),4))
print(round(accuracy(ssfc9),4))
}


and got the following output for the training set:

Now if I just use MSE and MAE, one response would be to use holt based on MSE, but MAE (or MASE) says use SES. Maybe too much info is not a good thing in this case. I also only gave you 12 months of data, but the set contains 24, which I roll through with a 12 month window. There are no zeros in the dataset, but for some reason Croston's method wins out in most cases. Totally confused.

• Is the table you provide computed from the rolling window or not? The code you show computes in-sample measures, which are not particularly useful for evaluating forecasting models. Commented Sep 12, 2018 at 12:08
• No, the table I provided is just for the first window, which would be used to determine the first forecast at h=1. Then it's rolled forward to determine the best method for the second and subsequent forecasts. Commented Sep 13, 2018 at 13:45
• If the table is actually computed with the code you've shown, it is not useful because those are in-sample measures (i.e. averaged over observations 1-12). You should use the data from 1-12 to forecast 13, then 1-13 to forecast 14, etc, then collect the forecast errors on 13-24 from each window, and compute the above summary measures over those. You may be getting good in-sample results from Croston's method simply because it's more complex, and overfitted. Commented Sep 13, 2018 at 14:17
• Full disclosure and I apologize that I didn't do this sooner, but I thought my question was straightforward. I'm evaluating software that is used to forecast inventory. It works by summing up all the demands from all locations in each month, developing a forecast model for the aggregate and then using that model on each location. I would like to determine how well the model (a) fits the aggregate demand and (b) how well a different model (chosen by the lowest error metric in my table) would affect the forecast at each location. Hence my original question. Commented Sep 13, 2018 at 17:21
• Your question is straightforward, but based on a false premise: you are not using "Rob Hyndman's approach for selecting the best model for forecasting" because you are looking at in-sample errors instead of out-of-sample errors. What you have in the table does not provide any information about forecasting performance. Commented Sep 13, 2018 at 17:30