Is it possible that a stacked LSTM for text generation is performing worse than a one layer LSTM?

Does anyone know if the generated text from a stacked LSTM is performing worse than a one layer LSTM? Is a possible answer that the Model is overfitting?

In my case after the first few epochs the text generated from a one layer LSTM makes much more sense than the text from stacked one (The los is also lower).

I used the code from the following link: https://github.com/rstudio/keras/blob/master/vignettes/examples/lstm_text_generation.R

And after some research I found that one could improve the model in adding layers to the LSTM model. I changed the model architecture to:

model %>%
layer_lstm(512, return_sequences = TRUE, input_shape = c(maxlen,
length(chars))) %>%
layer_dropout(0.2) %>%
layer_lstm(512, return_sequences = TRUE) %>%
layer_dropout(0.2) %>%
layer_lstm(512) %>%
layer_dense(length(chars)) %>%
layer_activation("softmax")


After checking the results from the first few epochs I realized that the text from the first model makes much more sense. How could this be explained?

Stacked LSTM:

 Epoch 2 loss 3.2485
diversity: 0.500000 ---------------

hof  ofnfif ffsoffffffssffofff feffollfl fhoffffflhffoefohfffhhof ofaffn
fefflffefssfnnsnf hlnowfnfnlgfsflfhffonnfhff
nfinffffhnhfffhffhfoffhfffannsh ff effffehhffghffrffinf fnffffefffffhfff
fnhd  fiflsfhfff hofhfhffefhfhf sffofhfefohhffffffff  fffffoho  ffffnh  iff
lf lfnfffffenffcffffffffwfffffsefff
fffffhffflfnsffnffffofnhfnfhhofflohoffnffnffhhcfhft ffflnnffnf
fffaynhfhfhfffshlhmoffhhfhnhfhff f


LSTM from the example:

 Epoch 2 loss 1.6497
diversity: 0.500000 ---------------

g leaght in them the man and the spirits of the senser the ling of an
almosing the somethy what is discrient of the most superfice the notion of
the most former some and to and he
will and a press to the sense, the spuring of his some proprease of the
more and be supoiled and evil and and what the supperance of the man
contination of such and of all the the takent of the simperfulned the
procratio

• I might have a clue. Can you put the accuracies you get from the two models? – BIM Aug 22 '18 at 15:31
• (1) It's generally true that it's harder to train a model with more parameters. Are you sure that the multi-layer model has converged? (2) Overfitting occurs when the training loss is dramatically lower than the out-of-sample loss. You haven't posted any information about your models, so we can't say whether this has occurred. – Sycorax Aug 22 '18 at 15:37
• I have edited my question. – Henryk Borzymowski Aug 22 '18 at 16:19
• I'm afraid that your edit doesn't really address the concerns that I raised in my comment. – Sycorax Aug 22 '18 at 16:21