I programmed a feedforward neural network for forecasting a time series, but the forecast is not stable and reasonable. I used a non-seasonal lag of 3, hence produced a gliding window of 3 as input for the NN, meaning 3 y-values of the series were given as input and the next one was labeled for training. The forecasting then was done in an autoregressive manner. My method was similar to making a AR(3) model without noise and drift. I am not sure the network results are better than a random walk though.
Specification of the network: 2 hidden layers; First layer with 12 units and relu activation. Second one with 8 units and also relu. Output was of course linear and one dimensional. As loss the MSE was used and the loss was optimized with ADAM. Epochs = 400, batch size = 2. The input data was MinMax scaled between $[0,1]$.The whole data has 202 samples.
The validation set was created by using
validation_split=0.15 from keras, which splits the last 15% of the training as test set. Both training and test were not shuffled to avoid temporal leaks.
I used R2 and MSE for goodness of fit. R2 training: 0.769, MSE training: 0.009, R2 test: 0.436, MSE test: 0.012
I tried different network topologies, optimizers and epochs. What surprises me is that this worked fine when using the
nnetar function in R. This function only has one hidden layer and when configuring the same number of lags it produces very good results. It uses the BFGS algorithm and as far as I could read the source code does not involve regularization (The function also averages over different trained networks, but I still get results when I don't average).
The code and data I worked with will be provided soon. (Not sure yet in which form to provide it here, since the code is about 100 lines).
For the color labels of the time series plot:
- Orange: The set used for training the NN
- Red: NN output for the training data after being trained
- Blue: The test data (NN has never seen this when being trained)
- Green: NN output for the test data after being trained.
- Voilet: The autoregressive forecast of the NN starting with the 3 last values of the test set for the first prediction. Then continue using the gliding window as explained