0
$\begingroup$

I am new to forecasting. Therefore, I do not sure many terms used in this process. There are original time-series data from 1980 Q1 until 2019 Q4. Then the data is split into the in-sample set (1980 Q1 - 2014 Q4) and the out-sample set (2015 Q1 - 2019 Q4). We have to make a forecast for the period from 2015 Q1 to 2019 Q4 using the in-sample set and then perform forecast accuracy. What does it mean by in-sample set and out-sample set? Does this mean that I have to test whether the forecast I made from the in-sample set is less error than the out-sample set given?

$\endgroup$
2
  • $\begingroup$ Basically you are splitting your dataset into two parts, an insample part (usually called training set) and an outsample part (usually called test set). You use the traininig set to train your predictive model, then you make predictions using your test set, and compare your predictions against the actual values on the test set. $\endgroup$ Apr 20, 2021 at 8:04
  • $\begingroup$ So that means in order to get the error measures, the out-sample values minus the predictions from the in-sample set? $\endgroup$ Apr 20, 2021 at 8:47

1 Answer 1

1
$\begingroup$

You fit your model (e.g., Exponential Smoothing) to the training sample, which is 1980Q1-2014Q4. Then you use it to forecast into the holdout sample, i.e., 2015Q1-2019Q4. Finally, you assess the accuracy of your forecast on the holdout sample, e.g., using the MSE, where you take the average of the squared errors between your forecast and the holdout actuals.

You will usually do this when you face a choice between different forecasting models. If method A performs better on the holdout sample than method B, the you hope that this will also be true when you use both methods on "real" data and forecast into the "true" future.

It is important to compare forecast accuracy out of sample, not in sample (also known as fit). Any method will try to fit its training data as closely as possible, and more complex methods can usually fit better. But that does not mean that they forecast better, because they may be overfitting.

So: always assess forecasts out of sample.

You may want to take a look at Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman or Forecasting: Principles and Practice (3rd ed.) by Athanasopoulos & Hyndman, in particular the section on forecast accuracy measurement.

$\endgroup$
1
  • $\begingroup$ As an example I chose between 8 models each month. I leave out the last 12 months and predict them with the rest of the data. Based on a MAPE I pick the best 3 models and then use this to project the next 2 years of data. I also use the hold out data set to select the ARIMA model (which is experimental for me, I use ESM models otherwise). $\endgroup$
    – user54285
    Apr 20, 2021 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.