0
$\begingroup$ 
library("forecast")
ss<-c(29,36,36,48,93,28,35,28,37,50,37,3,25,28,40,45,38,43,34,44,43,25,33,34)
ss<-ts(ss,f=12,start=c(2016,1))
for (i in 1:12){
ssfc <- ses(ss[c(i:(11+i))],h=1)
ssfc2 <- meanf(ss[c(i:(11+i))],h=1)
ssfc3 <- naive(ss[c(i:(11+i))],h=1)
ssfc4 <- snaive(ss[c(i:(11+i))],h=1)
ssfc5 <- rwf(ss[c(i:(11+i))],h=1,drift=TRUE)
ssfc6 <- croston(ss[c(i:(11+i))],h=1)
ssfc8 <- holt(ss[c(i:(11+i))],h=1)
ssfc9<-holt(ss[c(i:(11+i))],h=1,damped=TRUE)
print(round(accuracy(ssfc),4))
print(ssfc8[["model"]])
print(round(accuracy(ssfc2),4))
print(round(accuracy(ssfc3),4))
print(round(accuracy(ssfc4),4))
print(round(accuracy(ssfc5),4))
print(round(accuracy(ssfc6),4))
print(round(accuracy(ssfc8),4))
print(round(accuracy(ssfc9),4))
}

and got the following output for the training set:

enter image description here

Now if I just use MSE and MAE, one response would be to use holt based on MSE, but MAE (or MASE) says use SES. Maybe too much info is not a good thing in this case. I also only gave you 12 months of data, but the set contains 24, which I roll through with a 12 month window. There are no zeros in the dataset, but for some reason Croston's method wins out in most cases. Totally confused.

Improve this question
$\endgroup$
7
  • $\begingroup$ Is the table you provide computed from the rolling window or not? The code you show computes in-sample measures, which are not particularly useful for evaluating forecasting models. $\endgroup$
    – Chris Haug
    Commented Sep 12, 2018 at 12:08
  • $\begingroup$ No, the table I provided is just for the first window, which would be used to determine the first forecast at h=1. Then it's rolled forward to determine the best method for the second and subsequent forecasts. $\endgroup$
    – Angus
    Commented Sep 13, 2018 at 13:45
  • $\begingroup$ If the table is actually computed with the code you've shown, it is not useful because those are in-sample measures (i.e. averaged over observations 1-12). You should use the data from 1-12 to forecast 13, then 1-13 to forecast 14, etc, then collect the forecast errors on 13-24 from each window, and compute the above summary measures over those. You may be getting good in-sample results from Croston's method simply because it's more complex, and overfitted. $\endgroup$
    – Chris Haug
    Commented Sep 13, 2018 at 14:17
  • $\begingroup$ Full disclosure and I apologize that I didn't do this sooner, but I thought my question was straightforward. I'm evaluating software that is used to forecast inventory. It works by summing up all the demands from all locations in each month, developing a forecast model for the aggregate and then using that model on each location. I would like to determine how well the model (a) fits the aggregate demand and (b) how well a different model (chosen by the lowest error metric in my table) would affect the forecast at each location. Hence my original question. $\endgroup$
    – Angus
    Commented Sep 13, 2018 at 17:21
  • $\begingroup$ Your question is straightforward, but based on a false premise: you are not using "Rob Hyndman's approach for selecting the best model for forecasting" because you are looking at in-sample errors instead of out-of-sample errors. What you have in the table does not provide any information about forecasting performance. $\endgroup$
    – Chris Haug
    Commented Sep 13, 2018 at 17:30

1 Answer 1

Reset to default
1
$\begingroup$

  • Are you sure 12 rolling window tests warrant so many digits in your result table? In other words, what is the uncertainty on your error estimates and how does that compare to the observed differences? Statistically speaking, can you distinguish 19.40 from from 19.99?

  • Keep in mind: finding "those 4 methods perform well, it doesn't matter which of them we take" is a valid result. More than one method may be equally suitable.
    I don't work with time series, so I don't know how to interpret ses, meanf, holt and holt damped to achieve similar predictive performance. But with my spectroscopic data, if I found, say, that principal component regression and partial least squares do equally well, whereas OLS and neural network don't that would not be surprising. Interpretation would be that linear models are suitable, but some regularization is needed.

Improve this answer
$\endgroup$
2
  • $\begingroup$ These are good points and to answer your question, no, four decimal places are not warranted. It's just a format I chose. At best, only 2 are significant. There's still the issue of Croston's method that I can't understand, but thanks for your response. $\endgroup$
    – Angus
    Commented Sep 12, 2018 at 12:25
  • $\begingroup$ Well, do check the variance over your tests. I've seen quite a number of "optimizations" where not even the 1st digit was certain... Regression often is somewhat better behaved in that respect than classification, tough. $\endgroup$
    – cbeleites
    Commented Sep 12, 2018 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.