I'm carrying out a project of predicting time series data with an LSTM. I tried out the experiment three times with randomly sampled data(about 920,000 lines each)

I've stacked 3 layers of LSTM cells, used l1(0.01) regularization, used dropout, tried shuffling the dataset for every epoch, used ADAM optimizer..

but I get the error curve as follows, which seems to signify overfitting

x-axis : epochs

y-axis : error in terms of mean squared error

the blue line indicates the test set, and the orange train set

3 experiments

Can somebody give suggestions on what I should try? Maybe it's a matter on the dataset itself?

  • $\begingroup$ What do the plots show? Please label your data and explain what you think it means. $\endgroup$ – Sycorax Jun 17 '18 at 2:09
  • $\begingroup$ Yup, added the notations :) $\endgroup$ – HyeongGyu Froilan Choi Jun 17 '18 at 2:19
  • $\begingroup$ Why are there two lines? $\endgroup$ – Sycorax Jun 17 '18 at 2:27
  • $\begingroup$ the blue indicates the test set and the orange, train set $\endgroup$ – HyeongGyu Froilan Choi Jun 17 '18 at 2:29
  • $\begingroup$ Would you articulate how you made the validation set? What percent of the total data is it? Is it duplicated in the training set? Is there variables upon which it should selectively sub-sampled (stratified)? The first plot does not look like anything is failing. I have had NN's take 20k iterations to start getting close to reasonable fit, so 10 steps is really very few epochs. Also the error rates of interior layers may take longer to converge than edge weights because the information at the edge nodes takes longer to effectively diffuse inward. $\endgroup$ – EngrStudent Jun 28 '18 at 11:57

Yeah, that’s overfitting because the test error is much larger than the training error.

Three stacked LSTMs is hard to train. Try a simpler network and work up to a more complex one. Keep in mind that the tendency of adding LSTM layers is to grow the magnitude of the memory cells. Linked memory-forget cells enforce memory convexity and make it easier to train deeper LSTM networks.

Learning rate tweaking or even scheduling might also help.

In general, fitting a neural network involves a lot of experimentation and refinement. Finding the best network involves tuning a lot of dials together.

  • $\begingroup$ cool, I've never thought 3 stacked layers could be the cause of overfitting. I'd better try that out first. Thx a ton! $\endgroup$ – HyeongGyu Froilan Choi Jun 17 '18 at 3:04
  • $\begingroup$ More parameters means more model capacity and More model capacity can cause overfitting. $\endgroup$ – Sycorax Jun 17 '18 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.