What is in-sample and out-sample set in forecasting?

Question

I am new to forecasting. Therefore, I do not sure many terms used in this process. There are original time-series data from 1980 Q1 until 2019 Q4. Then the data is split into the in-sample set (1980 Q1 - 2014 Q4) and the out-sample set (2015 Q1 - 2019 Q4). We have to make a forecast for the period from 2015 Q1 to 2019 Q4 using the in-sample set and then perform forecast accuracy. What does it mean by in-sample set and out-sample set? Does this mean that I have to test whether the forecast I made from the in-sample set is less error than the out-sample set given?

Answer 1

You fit your model (e.g., Exponential Smoothing) to the training sample, which is 1980Q1-2014Q4. Then you use it to forecast into the holdout sample, i.e., 2015Q1-2019Q4. Finally, you assess the accuracy of your forecast on the holdout sample, e.g., using the MSE, where you take the average of the squared errors between your forecast and the holdout actuals.

You will usually do this when you face a choice between different forecasting models. If method A performs better on the holdout sample than method B, the you hope that this will also be true when you use both methods on "real" data and forecast into the "true" future.

It is important to compare forecast accuracy out of sample, not in sample (also known as fit). Any method will try to fit its training data as closely as possible, and more complex methods can usually fit better. But that does not mean that they forecast better, because they may be overfitting.

So: always assess forecasts out of sample.

You may want to take a look at Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman or Forecasting: Principles and Practice (3rd ed.) by Athanasopoulos & Hyndman, in particular the section on forecast accuracy measurement.