Does anyone know if the generated text from a stacked LSTM is performing worse than a one layer LSTM? Is a possible answer that the Model is overfitting?
In my case after the first few epochs the text generated from a one layer LSTM makes much more sense than the text from stacked one (The los is also lower).
I used the code from the following link: https://github.com/rstudio/keras/blob/master/vignettes/examples/lstm_text_generation.R
And after some research I found that one could improve the model in adding layers to the LSTM model. I changed the model architecture to:
model %>% layer_lstm(512, return_sequences = TRUE, input_shape = c(maxlen, length(chars))) %>% layer_dropout(0.2) %>% layer_lstm(512, return_sequences = TRUE) %>% layer_dropout(0.2) %>% layer_lstm(512) %>% layer_dense(length(chars)) %>% layer_activation("softmax")
After checking the results from the first few epochs I realized that the text from the first model makes much more sense. How could this be explained?
Epoch 2 loss 3.2485 diversity: 0.500000 --------------- hof ofnfif ffsoffffffssffofff feffollfl fhoffffflhffoefohfffhhof ofaffn fefflffefssfnnsnf hlnowfnfnlgfsflfhffonnfhff nfinffffhnhfffhffhfoffhfffannsh ff effffehhffghffrffinf fnffffefffffhfff fnhd fiflsfhfff hofhfhffefhfhf sffofhfefohhffffffff fffffoho ffffnh iff lf lfnfffffenffcffffffffwfffffsefff fffffhffflfnsffnffffofnhfnfhhofflohoffnffnffhhcfhft ffflnnffnf fffaynhfhfhfffshlhmoffhhfhnhfhff f
LSTM from the example:
Epoch 2 loss 1.6497 diversity: 0.500000 --------------- g leaght in them the man and the spirits of the senser the ling of an almosing the somethy what is discrient of the most superfice the notion of the most former some and to and he will and a press to the sense, the spuring of his some proprease of the more and be supoiled and evil and and what the supperance of the man contination of such and of all the the takent of the simperfulned the procratio