Newbie question, sorry. I have a highly seasonal monthly time series, predictable with no exogenous/independent variables and no obvious trend. I want to show that a suitable state space model (using dlm) is a better predictor than both Holt Winters and a SARIMA model and that some more modern methods (single layer neural network nnetarfrom the R package forecast and XGBoost) are better still.

1) I have split my data 80/20 train/test, what single measure should I use to establish a comparative measure of predictive power across all these methods? A data science competition I saw recently used sum of log squares:

$$\sum (log(predict)-log(test))^2$$

Is this appropriate across all methods here? I've not seen the log correction before, what's the purpose of this?

2) I have seen some literature suggesting a 60/20/20 split of data to allow for cross validation. Why wouldn't I just fit the model to 80% of my data and use part 1) to select the best method?

  • $\begingroup$ Regarding 1, it depends on your loss function. There are many valid measures, but they make sense for different loss functions. Regarding 2, you need to do parameter tuning in some methods, and there you need cross validation. How would you tune parameters otherwise? $\endgroup$ – Richard Hardy May 1 '17 at 5:34
  • $\begingroup$ It's unlikely that nntetar or xgb will have better forecast accuracy than an ETS or ARIMA model for a univariate time series. Generally speaking, they require extreme amounts of data and many exogenous variables to beat "traditional" time series methods. $\endgroup$ – AnscombesGimlet May 2 '17 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.