2
$\begingroup$

I'm using Keras with Theano backend (for results reproducibility) which aims to give a one step ahead of a time series. It takes lagged sequences from 16 different time series (one of which is the target time series) and fits a network with an initial Dense Layer, then a GRU layer and finally a Dense Layer too receive one output. I have approximately 650 data values for each time series training set. I've normalised each of the time series to the range $(0,1)$, just using my training data to fit the MinMaxScaler from sckitlearn. My issue is that my training loss drops dramatically within the first few epochs and then plateaus, irrespective of the number of epochs I use to train. The value that the loss plateaus at is fairly high, a MSE of around 2 for 400 epochs, for target values of around 10. It's my belief that the training set should reduce on average every epoch. In the graph below, I train for 400 epochs and I use a simple hold out validation set representing the last 10% of the training set, rather than a full cross validation at the moment, so it is not alarming that the validation loss is less than the training. The test legend refers to the validation set. Does anyone have some inclination of why model cannot learn the training data very well? enter image description here

Also on an (potentially) unrelated note, my model consistently seems to over and underestimate the true values but learns the overall pattern of the data, see below.enter image description here

If anyone has any suggestions on why my model doesn't seem to perform/learn well, or suggestions to improve it I'd be very grateful. Thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ Have you tried adjusting your network architecture? Hyperparameter optimization? Why GRU, as opposed to LSTM? $\endgroup$
    – redress
    Commented Jun 6, 2017 at 21:32
  • $\begingroup$ In terms of adjusting the network, I have tried adding in a second layer of both GRU and LSTM and optimised the hidden layers, batch size, sequence length, activation choice and dropout value. When running this optimisation I aim to reduce the cross validated error from a walk forward CV. The above graphs are then found by using the optimised hyper parameters but only with a standard holdout validation set for speed. $\endgroup$
    – George1811
    Commented Jun 6, 2017 at 21:45

2 Answers 2

Reset to default
3
$\begingroup$

You arent running into any glaring issues, and at a certain point it might be wise to accept the business value of your model and not waste 30% of your time on 5% of your error.

If you are really worried about the performance, you can always run you model through an AutoML framework like TPOT and see if their Bayesian Optimization finds something that you weren't able to manually.

Improve this answer
$\endgroup$
0
1
$\begingroup$

True data vs Prediction plot clearly shows the network is acting like a naïve predictor... it is simply taking the t-1 value and outputting something very close at t.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This seems to be the case since the shape of the prediction seems to look like the true value one step ahead. But the actual magnitude of the prediction is on a few occasions somewhat different in magnitude. $\endgroup$
    – Michael R. Chernick
    Commented Jun 8, 2017 at 1:37
  • $\begingroup$ Is there any way to combat this without getting more data? Maybe including data which are linear combinations of others? $\endgroup$
    – George1811
    Commented Jun 8, 2017 at 13:47
  • $\begingroup$ Look at this discussion of the same issue on Matlab: mathworks.com/matlabcentral/answers/… Be sure to open and look at the older, hidden replies there too. $\endgroup$
    – Molasar
    Commented Jun 9, 2017 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.