Does anyone know if the generated text from a stacked LSTM is performing worse than a one layer LSTM? Is a possible answer that the Model is overfitting?

In my case after the first few epochs the text generated from a one layer LSTM makes much more sense than the text from stacked one (The los is also lower).

I used the code from the following link: https://github.com/rstudio/keras/blob/master/vignettes/examples/lstm_text_generation.R

And after some research I found that one could improve the model in adding layers to the LSTM model. I changed the model architecture to:

model %>%
layer_lstm(512, return_sequences = TRUE, input_shape = c(maxlen, 
length(chars))) %>%
layer_dropout(0.2) %>%
layer_lstm(512, return_sequences = TRUE) %>%
layer_dropout(0.2) %>%
layer_lstm(512) %>%
layer_dense(length(chars)) %>%

After checking the results from the first few epochs I realized that the text from the first model makes much more sense. How could this be explained?

Stacked LSTM:

 Epoch 2 loss 3.2485
 diversity: 0.500000 ---------------

 hof  ofnfif ffsoffffffssffofff feffollfl fhoffffflhffoefohfffhhof ofaffn 
 fefflffefssfnnsnf hlnowfnfnlgfsflfhffonnfhff 
 nfinffffhnhfffhffhfoffhfffannsh ff effffehhffghffrffinf fnffffefffffhfff 
 fnhd  fiflsfhfff hofhfhffefhfhf sffofhfefohhffffffff  fffffoho  ffffnh  iff 
 lf lfnfffffenffcffffffffwfffffsefff 
 fffffhffflfnsffnffffofnhfnfhhofflohoffnffnffhhcfhft ffflnnffnf 
 fffaynhfhfhfffshlhmoffhhfhnhfhff f

LSTM from the example:

 Epoch 2 loss 1.6497
 diversity: 0.500000 ---------------

 g leaght in them the man and the spirits of the senser the ling of an
 almosing the somethy what is discrient of the most superfice the notion of 
 the most former some and to and he
 will and a press to the sense, the spuring of his some proprease of the 
 more and be supoiled and evil and and what the supperance of the man 
 contination of such and of all the the takent of the simperfulned the 
  • $\begingroup$ I might have a clue. Can you put the accuracies you get from the two models? $\endgroup$ – BIM Aug 22 '18 at 15:31
  • 1
    $\begingroup$ (1) It's generally true that it's harder to train a model with more parameters. Are you sure that the multi-layer model has converged? (2) Overfitting occurs when the training loss is dramatically lower than the out-of-sample loss. You haven't posted any information about your models, so we can't say whether this has occurred. $\endgroup$ – Sycorax Aug 22 '18 at 15:37
  • $\begingroup$ I have edited my question. $\endgroup$ – Henryk Borzymowski Aug 22 '18 at 16:19
  • $\begingroup$ I'm afraid that your edit doesn't really address the concerns that I raised in my comment. $\endgroup$ – Sycorax Aug 22 '18 at 16:21

You've posted loss values from the second epoch of each network. I wouldn't read too much into the difference, since the loss of the 2-layer network at the second epoch is clearly not converged (it's 3.2, which is enormous for this flavor of language models). With loss that high, it's absolutely typical that the text it generates is nonsense.

I recommend training the 2-layer network until its loss equals or beats the loss of the 1-layer network. It's generally true that it's harder to train a model with more parameters, so this may require some effort to achieve: tweaking the optimizer and learning rate, batch size, sequence lengths and so on.

The primary benefit to a 2-layer LSTM network in this context is that it has more capacity to learn the language; in other words, the minimum loss that it obtains may be substantially lower than the minimum loss of the 1-layer network. On the other hand, this increase in complexity comes at the cost of increased computation time, and a harder optimization problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.