14
$\begingroup$

I have several intermittent data. Based on those data, I would like to compare several forecasting methods (Exponential Smoothing, Moving Average, Croston, and Syntetos-Boylan), and decide whether Croston or Syntetos Boylan is better than SES or MA for intermittent data or not. The measure I'd like to compare is Mean Absolute Rate or Mean Squared Rate proposed by Kourentzes (2014) instead of usual MAPE, MSE measure, at every level of \alpha$ smoothing parameter, assuming that the smoothing parameter used for Inter demand Interval and demand size in Croston and Syntetos boylan is the same.

My question is: 1. Considering that for every data, there is the possibility that the value of optimal alpha is different for each smoothing methods, therefore a value of alpha in a method may minimize the the MAR or MSR and may not in other method, such that that one method may be better than other method for that value of alpha and may not in other method. How does one solve this kind of problem? my current solution is to compare the two graph of MAR for every level of alpha between two different method. my expectation is that the two different method will show different characteristics when profile analysis is done.

  1. Is there any better solution, like experimental design? I am rather confused in how to design the experiments. the observation is those several data, the level is smoothing parameter alpha, treatment is those methods. and the value is the MAR. is it viable? and logical to do? The hypothesis is whether there is differences in each treatment at every level of alpha or not. and I doubt the linearity assumption is fulfilled here.

Edit: Anyway, I do not think this is viable as research question. The fact that the error measure is scale dependent (if my definition of scale dependent are right) made the study of this problem is quite problematic, as there is no way to compare the different methods of forecasting.

$\endgroup$
4
  • 1
    $\begingroup$ have you considered out of sample/hold out testing of your predictions from different forecasting methods? $\endgroup$
    – forecaster
    Sep 10, 2014 at 2:44
  • $\begingroup$ I do not know if there is meaning to hold out of sample data in Croston method. as far as I know, croston and syntetos boylan forecasting had to be evaluated everytime there is a demand. therefore, evaluating the forecast by holding out of sample data seems strange to me. $\endgroup$
    – Fikri
    Sep 10, 2014 at 3:46
  • $\begingroup$ yes you can use hold out data for intermittent data. That is the only way you can be sure if your model parameters are reliable. If you small data then I would use cross validation or jack knife methos. In this article the authors have compared forecasting methods using initialization and test (hold out) data for intermittent forecasting and Croston method is one of them. $\endgroup$
    – forecaster
    Sep 10, 2014 at 15:35
  • $\begingroup$ Did you google "forecast comparison"? There's a lot of research in this field in econometrics since at least 1980s $\endgroup$
    – Aksakal
    Sep 17, 2015 at 14:09

1 Answer 1

1
$\begingroup$

Model: $y_{t+1}=f(y_{0},\ldots,y_{t}, \overrightarrow{a})+\epsilon_{t}$

$\overrightarrow{a}$ is a vector of model parameters

What you are currently proposing is essentially:

  1. For each model $f(y_{0},\ldots,y_{t}, \overrightarrow{a})$ use some sort of Least Squares / NLS to fit a model (find $\overrightarrow{a}$)
  2. Choose an $\alpha$ value
  3. Use some function $g(f(y_{0},\ldots,y_{t}, \overrightarrow{a}),\alpha)$ to evaluate the models.
  4. If optimal model depends importantly on $\alpha$ then ????

Can you do endogenous sampling? If so, how about directly estimating optimal (fix maximizing) functions $g(f(y_{0},\ldots,y_{t}, \overrightarrow{a}),\alpha)$ directly for multiple values of $\alpha$. You could take the models and run them in parallel, making a family of predictions. You could then increase the sampling likelihood when the models disagreed particularly in their predictions. This would increase the informativeness of the limited sampling in distinguishing between models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.