# Recurrent Neural Network Training Loss does not decrease past a certain value

I'm using Keras with Theano backend (for results reproducibility) which aims to give a one step ahead of a time series. It takes lagged sequences from 16 different time series (one of which is the target time series) and fits a network with an initial Dense Layer, then a GRU layer and finally a Dense Layer too receive one output. I have approximately 650 data values for each time series training set. I've normalised each of the time series to the range $(0,1)$, just using my training data to fit the MinMaxScaler from sckitlearn. My issue is that my training loss drops dramatically within the first few epochs and then plateaus, irrespective of the number of epochs I use to train. The value that the loss plateaus at is fairly high, a MSE of around 2 for 400 epochs, for target values of around 10. It's my belief that the training set should reduce on average every epoch. In the graph below, I train for 400 epochs and I use a simple hold out validation set representing the last 10% of the training set, rather than a full cross validation at the moment, so it is not alarming that the validation loss is less than the training. The test legend refers to the validation set. Does anyone have some inclination of why model cannot learn the training data very well?

Also on an (potentially) unrelated note, my model consistently seems to over and underestimate the true values but learns the overall pattern of the data, see below.

If anyone has any suggestions on why my model doesn't seem to perform/learn well, or suggestions to improve it I'd be very grateful. Thanks

• Have you tried adjusting your network architecture? Hyperparameter optimization? Why GRU, as opposed to LSTM? Jun 6 '17 at 21:32
• In terms of adjusting the network, I have tried adding in a second layer of both GRU and LSTM and optimised the hidden layers, batch size, sequence length, activation choice and dropout value. When running this optimisation I aim to reduce the cross validated error from a walk forward CV. The above graphs are then found by using the optimised hyper parameters but only with a standard holdout validation set for speed. Jun 6 '17 at 21:45