How to compare forecasting methods?

Question

I have several intermittent data. Based on those data, I would like to compare several forecasting methods (Exponential Smoothing, Moving Average, Croston, and Syntetos-Boylan), and decide whether Croston or Syntetos Boylan is better than SES or MA for intermittent data or not. The measure I'd like to compare is Mean Absolute Rate or Mean Squared Rate proposed by Kourentzes (2014) instead of usual MAPE, MSE measure, at every level of \alpha$ smoothing parameter, assuming that the smoothing parameter used for Inter demand Interval and demand size in Croston and Syntetos boylan is the same.

My question is: 1. Considering that for every data, there is the possibility that the value of optimal alpha is different for each smoothing methods, therefore a value of alpha in a method may minimize the the MAR or MSR and may not in other method, such that that one method may be better than other method for that value of alpha and may not in other method. How does one solve this kind of problem? my current solution is to compare the two graph of MAR for every level of alpha between two different method. my expectation is that the two different method will show different characteristics when profile analysis is done.

2. Is there any better solution, like experimental design? I am rather confused in how to design the experiments. the observation is those several data, the level is smoothing parameter alpha, treatment is those methods. and the value is the MAR. is it viable? and logical to do? The hypothesis is whether there is differences in each treatment at every level of alpha or not. and I doubt the linearity assumption is fulfilled here.

Edit: Anyway, I do not think this is viable as research question. The fact that the error measure is scale dependent (if my definition of scale dependent are right) made the study of this problem is quite problematic, as there is no way to compare the different methods of forecasting.

Answer 1

Model: $y_{t+1}=f(y_{0},\ldots,y_{t}, \overrightarrow{a})+\epsilon_{t}$

$\overrightarrow{a}$ is a vector of model parameters

What you are currently proposing is essentially:

1. For each model $f(y_{0},\ldots,y_{t}, \overrightarrow{a})$ use some sort of Least Squares / NLS to fit a model (find $\overrightarrow{a}$)
2. Choose an $\alpha$ value
3. Use some function $g(f(y_{0},\ldots,y_{t}, \overrightarrow{a}),\alpha)$ to evaluate the models.
4. If optimal model depends importantly on $\alpha$ then ????

Can you do endogenous sampling? If so, how about directly estimating optimal (fix maximizing) functions $g(f(y_{0},\ldots,y_{t}, \overrightarrow{a}),\alpha)$ directly for multiple values of $\alpha$. You could take the models and run them in parallel, making a family of predictions. You could then increase the sampling likelihood when the models disagreed particularly in their predictions. This would increase the informativeness of the limited sampling in distinguishing between models.