Time Series Forecast with Neural Networks blows up

Question

I programmed a feedforward neural network for forecasting a time series, but the forecast is not stable and reasonable. I used a non-seasonal lag of 3, hence produced a gliding window of 3 as input for the NN, meaning 3 y-values of the series were given as input and the next one was labeled for training. The forecasting then was done in an autoregressive manner. My method was similar to making a AR(3) model without noise and drift. I am not sure the network results are better than a random walk though.

Specification of the network: 2 hidden layers; First layer with 12 units and relu activation. Second one with 8 units and also relu. Output was of course linear and one dimensional. As loss the MSE was used and the loss was optimized with ADAM. Epochs = 400, batch size = 2. The input data was MinMax scaled between $[0,1]$.The whole data has 202 samples.

The validation set was created by using validation_split=0.15 from keras, which splits the last 15% of the training as test set. Both training and test were not shuffled to avoid temporal leaks.

I used R2 and MSE for goodness of fit. R2 training: 0.769, MSE training: 0.009, R2 test: 0.436, MSE test: 0.012

I tried different network topologies, optimizers and epochs. What surprises me is that this worked fine when using the nnetar function in R. This function only has one hidden layer and when configuring the same number of lags it produces very good results. It uses the BFGS algorithm and as far as I could read the source code does not involve regularization (The function also averages over different trained networks, but I still get results when I don't average).

The code and data I worked with will be provided soon. (Not sure yet in which form to provide it here, since the code is about 100 lines).

For the color labels of the time series plot:

• Orange: The set used for training the NN
• Red: NN output for the training data after being trained
• Blue: The test data (NN has never seen this when being trained)
• Green: NN output for the test data after being trained.
• Voilet: The autoregressive forecast of the NN starting with the 3 last values of the test set for the first prediction. Then continue using the gliding window as explained

Answer 1

I am not an expert in NN but was intrigued by your question.I took your 202 sequential values with known periodicity of 19 into AUTOBOX my tool of choice and proceeded to automatically analyze the data. It yielded the following model suggesting a level shift at time period 152 in addition to a few pulses.

The Actual/Fit and Forecast graph as here

with cleansed data here and reasonable forecasts here . The plot of the model's residuals are here with acf here

Note also that two change points were detected in the error variance suggesting increased error variance at 56 and a reduction in error variance at 92 suggesting the need for Weighted Estimation (GLS) to render the error process homogeneous

This all suggests to me that your simulation incorporated a deterministic (and ignored) component which might explain why your forecast is flawed as you only included stochastic memory (arima) while the data included additional deterministic effects and some error variance changes.

Do you concur with this ? Do you have any reason why your simulation may have introduced these additional effects NOT remediable with NN bit clearly in the historical .

To continue my research in this NN vs Robust Time Series I used your suggestion of 15% and retained 32 values from the end and proceeded to develop the following model and forecasts for the out-of-sample 32 values.

This is what was obtained using 172 actuals and 32 retained.

and and .. no level shift at 152 due to only 20 values being available.

and with a 1.2% MAPE for the 32 periods.

I hope this helps you and others to clear up mysteries regarding NN and the possible analyses that are routinely available .

Answer 2

Based on the graph you posted, I would say you are falling into the "my fancy neural network is really just a random walk with drift" trap. If you look closely, you will notice that your forecasts are shifted by one step from the actuals: Your neural network just learned to copy the last value it saw and use it as the most likely next value $\pm$ a noise component (which is why it isn't an exact copy of the last value).

I don't have an exact answer, but here are two considerations:

• You said you had better results with NNETAR with only one hidden layer? So why are you using 2 hidden layers then? That will only lead to overfitting. Unless you are modeling a really complex and/or multivariate time series, I do not recommend using multiple hidden layers for a forecasting problem.

• Try training on multiple steps ahead instead of one step ahead, in my experience that led to better results and generally avoided the "random walk trap" which is so common when dealing with neural networks for forecasting.