2
$\begingroup$

I'm a bit confused about the process order in doing proper time series analysis/forecasting. Is it:

  1. Stationary/seasonality checks, do any transformations required
  2. Candidate model selection using ACF, PACF, and auto.arima... using all data
  3. Split data into training and test
  4. Choose best model based off accuracy measures (RMSE...) and Information Criteria (AICc, BIC..)
  5. etc

or:

  1. Stationary/seasonality checks, do any transformations required
  2. Split data into training and test
  3. Candidate model selection using ACF, PACF, and auto.arima... using just training data
  4. Choose best model based off accuracy measures (RMSE...) and Information Criteria (AICc, BIC..)
  5. etc

Or neither. I've looked around for examples and read through textbooks but can't find a straight answer.

First post so sorry if this isn't the kind of thing that should be asked here/if it was not asked correctly.

Thanks!

$\endgroup$
2
  • 1
    $\begingroup$ An introductory textbook to start with, Hyndman-Athanasopoulos Forecasting: Principles and Practice. $\endgroup$ Sep 11, 2021 at 5:26
  • $\begingroup$ Thanks! I've consulted that book before, and it seems to suggest I split the data first. $\endgroup$
    – Kyle Mouly
    Sep 11, 2021 at 16:48

1 Answer 1

2
$\begingroup$

I would second the recommendation for Hyndman-Athanasopoulos and note that there is a third edition (comment by Mehmet linked to second edition).

Time series forecasting is more of an art than a science sometimes but in general the second pipeline you described is better. The #1 pitfall with time series is look-ahead bias. This is where you use information to make a prediction which wouldn't have actually been available to you at the time.

Another related form of look-ahead bias is selecting features/models using testing data and so potentially overfitting to relationships which happen to hold true in the testing period.

You want to build features based on relationships learned in training and then see whether those relationships continue to hold true in the testing period. Even better you could have train/validation/testing periods for feature selection/model selection/model testing respectively.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.