11
$\begingroup$

I've been working for months on short-term load forecasting and the use of climate/weather data to improve the accuracy. I have a computer science background and for this reason I'm trying to not make big mistakes and unfair comparisons working with statistics tools like ARIMA models. I'd like to know your opinion about a couple of things:

  1. I'm using both (S)ARIMA and (S)ARIMAX models to investigate the effect of weather data on forecasting, do you think it would be necessary to use also Exponential Smoothing methods?

  2. Having a time series of 300 daily samples I'm starting from the first two weeks and I perform a 5 days-ahead forecast using models built with auto.arima R function (forecast package). Then, I add another sample to my data set and I calibrate again the models and I perform another 5 days forecast and so on until the end of the available data. Do you think this way to operate is correct?

Thank you for your suggestions, although the target of our work is an engineering journal article, I would like to do a work as rigorous as possible from a statistical point of view.

Improve this question
$\endgroup$
2

1 Answer 1

Reset to default
10
$\begingroup$

  1. I think it would be worth exploring exponential smoothing models as well. Exponential smoothing models are a fundamentally different class of models from ARIMA models, and may yield different results on your data.

  2. This sounds like a valid approach, and is very similar to the time series cross-validation method proposed by Rob Hyndman.

I would aggregate the cross-validation error from each forecast (exponential smoothing, ARIMA, ARMAX) and then use the overall error to compare the 3 methods.

You may also want to consider a "grid search" for ARIMA parameters, rather than using auto.arima. In a grid search, you would explore each possible parameter for an arima model, and then select the "best" ones using forecast accuracy.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for the reply, I'm using auto.arima with STEPWISE search disabled and I think that in this way it will explore all the parameters between the min-max range (I haven't read the Hyndman & Kandahar 2008 paper yet) $\endgroup$
    – Matteo De Felice
    Commented Oct 19, 2011 at 12:53
  • $\begingroup$ @Matteo De Felice: the thing is, it is optimizing those parameters based on AIC. I was suggesting it might be worth optimizing those parameters by their out-of-sample performance, which you could assess by cross-validation. Furthermore, if you want to compare other models (such as ETS), you will need an out-of-sample performance metric, as you can't compare AIC between different types of models (such as auto.arima and ets). $\endgroup$
    – Zach
    Commented Oct 19, 2011 at 13:24
  • $\begingroup$ at this moment I performed a grid search (using auto.arima with stepwise = FALSE) and then I tried the most frequent models in order to evaluate their performances. $\endgroup$
    – Matteo De Felice
    Commented Oct 25, 2011 at 11:38
  • 1
    $\begingroup$ @Matteo De Felice: If you're having trouble implementing time-series cross-validation, this blog post might help: robjhyndman.com/researchtips/tscvexample $\endgroup$
    – Zach
    Commented Dec 9, 2011 at 22:02
  • 1
    $\begingroup$ @Wart "the non-linear exponential smoothing models have no equivalent ARIMA counterparts. There are also many ARIMA models that have no exponential smoothing counterparts." $\endgroup$
    – Zach
    Commented Dec 13, 2017 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.