I have several intermittent data. Based on those data, I would like to compare several forecasting methods (Exponential Smoothing, Moving Average, Croston, and Syntetos-Boylan), and decide whether Croston or Syntetos Boylan is better than SES or MA for intermittent data or not. The measure I'd like to compare is Mean Absolute Rate or Mean Squared Rate proposed by Kourentzes (2014) instead of usual MAPE, MSE measure, at every level of \alpha$ smoothing parameter, assuming that the smoothing parameter used for Inter demand Interval and demand size in Croston and Syntetos boylan is the same.
My question is: 1. Considering that for every data, there is the possibility that the value of optimal alpha is different for each smoothing methods, therefore a value of alpha in a method may minimize the the MAR or MSR and may not in other method, such that that one method may be better than other method for that value of alpha and may not in other method. How does one solve this kind of problem? my current solution is to compare the two graph of MAR for every level of alpha between two different method. my expectation is that the two different method will show different characteristics when profile analysis is done.
- Is there any better solution, like experimental design? I am rather confused in how to design the experiments. the observation is those several data, the level is smoothing parameter alpha, treatment is those methods. and the value is the MAR. is it viable? and logical to do? The hypothesis is whether there is differences in each treatment at every level of alpha or not. and I doubt the linearity assumption is fulfilled here.
Edit: Anyway, I do not think this is viable as research question. The fact that the error measure is scale dependent (if my definition of scale dependent are right) made the study of this problem is quite problematic, as there is no way to compare the different methods of forecasting.