I want to evaluate several forecasting methods on the taylor time series using cross-validation.

How do I go about selecting the hyper-parameters for the methods? Should I manually select them ahead using fit statistics on another hold-out set? Or should I devise a system to re-select them for each train/test split in my CV?


  • for my dynamic harmonic regression forecast, do I pre-select the fourier terms order? do I pre-select the ARIMA order for my regression errors? or do I re-select them for every train/test split with auto-arima and a loop that minimizes AICc for several options of fourier terms order?
  • for my stl-decomposition based forecast, do I manually pre-select the s.window by looking at plots of a hold-out set? Or should I devise a way to re-select it for every train/test split? (how? by testing stationarity of residuals?)

r code (forecast library):


# should I select these ahead? or re-select them for each train/test split in my CV?
best.s.window <- 5
best.K <- c(9,5)
best.arima.order <- c(5,1,3)

methods.list <- list(
  rwf=rwf, snaive=snaive, meanf=meanf,
  stl=function(y, ...){stlf(y, s.window=best.s.window, ...)},
  dynamic.regression=function(y, ...){
    fit <- Arima(y, order=best.arima.order, xreg = fourier(y, K=best.K), lambda = 0)
    fcast <- forecast(fit, xreg=fourier(y, K=best.K, h=1), h=1)

lapply(methods.list, function(method) {
  taylor %>% subset(end=4*48*7) %>% tsCV(forecastfunction = method, window=3*48*7, h=1)
}) -> cv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.