I'm using Keras with Theano backend (for results reproducibility) which aims to give a one step ahead of a time series. It takes lagged sequences from 16 different time series (one of which is the target time series) and fits a network with an initial Dense Layer, then a GRU layer and finally a Dense Layer too receive one output. I have approximately 650 data values for each time series training set. I've normalised each of the time series to the range $(0,1)$, just using my training data to fit the MinMaxScaler from sckitlearn. My issue is that my training loss drops dramatically within the first few epochs and then plateaus, irrespective of the number of epochs I use to train. The value that the loss plateaus at is fairly high, a MSE of around 2 for 400 epochs, for target values of around 10. It's my belief that the training set should reduce on average every epoch. In the graph below, I train for 400 epochs and I use a simple hold out validation set representing the last 10% of the training set, rather than a full cross validation at the moment, so it is not alarming that the validation loss is less than the training. The test legend refers to the validation set. Does anyone have some inclination of why model cannot learn the training data very well?
If anyone has any suggestions on why my model doesn't seem to perform/learn well, or suggestions to improve it I'd be very grateful. Thanks