Choosing the right size of an out of sample data

Question

I am beginner in forecasting, especially forecasting with R and I am really willing to improve my knowledge.

Recently, I started practicing electricity consumption time series forecasting.

The first barrier I faced is the choice of out of sample data for assessing the forecast accuracy of the forecast model i will be using (regression with ARIMA errors).

I have data for 147 months and I want to forecast the next 24 months, for the period June 2013 to January 2015. Furthermore, I have read in @RobHyndman's online text book

Hyn­d­man, R.J. and Athana­sopou­los, G. (2013),
Forecasting: principles and practice,
(accessed 28 May 2013), section 2.5
under 'Training and test sets', that:

size of the test set is typ­i­cally about 20% of the total sam­ple, although this value depends on how long the sam­ple is and how far ahead you want to fore­cast

If I divide the dataset with 20% of it being the out-of-sample data, the forecast model applied to the in-sample data is not quite accurate, since I guess it fails to capture the recent trend, (which began in the middle of the last year), of decreased electricity consumption due to a significantly raised electricity tariff.

What do I do about this?

Can you possibly give me instruction on what would be considered appropriate size of the out of sample data. I also tried with 7 months of out-of-sample data, but I am afraid that there might be an overfitting issue. Is that right?

Answer 1

Not really my area of expertise but I think one answer should be “Nothing!” You could of course try to improve the model or try other techniques (but if you begin tuning the model on the basis of its performance in the test set, you still run the risk of “overfitting”) but changing the size of the test set does not seem to directly address this problem.

If today was still 2011 and we were trying to predict electricity consumption until 2013, this model would give us some seriously misleading predictions. This is precisely the type of things out-of-sample evaluation is supposed to pick up. You can look at it retrospectively today and interpret it as a trend that started last year because of some change to the electricity market but the conclusion remains the same: This model did not allow you to see it coming.

Also, if you read the sentence carefully you will notice that Rob Hyndman also stresses that size of the test set should depend on how far ahead you want to forecast. Intuitively, if you want to predict 24 months, a test set of 7 months is too short, no matter whether you have 100 or 10000 months of past data. For example, a good model of seasonal changes in your data could look very good even if it is unable to predict any year-on-year trend.

Answer 2

20% of 147 months is 30 months, which means that at you have over 2 years of data. If the trend began in the middle of last year, then it exists within that 2 year time frame.

If you feel that your forecast is not capturing your trend, then it is probably because you smoothed too much. If you used the average instead of the drift or seasonal method, then it unlikely went uncaptured. The problem isn't with your sample size but with your method of forecasting - and specifically how much smoothing you are doing.