# LSTM good at hallucinating, useless at ground truth prediction?

I was interested in this project, so I cloned it and trained it on Moby Dick, for this challenge. The goal is to predict the next character given the past ground-truth characters. Overfitting is not an issue, if the neural network remembers actual passages, the better.

After training the neural network, which has the following structure:

model = Sequential()
model.compile(loss="categorical_crossentropy", optimizer="rmsprop")

If we now feed one character to the neural network, take whatever it predicts and feed everything it has produced so far back into repeatedly, it hallucinates reasonable looking text:

and cold, Sabayow, dreaming is not neck to comes to my fing that foolish a queer part of a harpoon from the sea, vessel in ite harpooneers, sharkless and round by all his occasional things. But no more of a nose, ether have inditably upon the one city of the Sperm Whale; which the land ships, grieving whales, are perhaps, at the barroo arp then, the transparent shoul I felt upon discreding a compass. Nor did the Captain Black and a ghatel story is absunding his chiefims.

Done as such:

def generate_text(model, length, vocab_size, ix_to_char):
# starting with random character
ix = [np.random.randint(vocab_size)]
y_char = [ix_to_char[ix[-1]]]
X = np.zeros((1, length, vocab_size))
for i in range(length):
# appending the last predicted character to sequence
X[0, i, :][ix[-1]] = 1
print(ix_to_char[ix[-1]], end="")
ix = np.argmax(model.predict(X[:, :i+1, :])[0], 1)
y_char.append(ix_to_char[ix[-1]])
return ('').join(y_char)

So now I thought I could feed it a sequence of ground truth, and figure that it'd give reasonable estimates for what is coming up next:

class Predictor:
def __init__(self, model, vocab_size, ix_to_char):
self.model = model
self.vocab_size = vocab_size
self.ix_to_char = ix_to_char
self.char_to_ix = {v:k for k, v in ix_to_char.items()}
self.past = []

def predict(self, last_c):
self.past.append(self.char_to_ix[last_c])
X = np.zeros((1, len(self.past), self.vocab_size))
for i in range(len(self.past)):
X[0, i, self.past[i]] = 1.
ix = np.argmax(self.model.predict(X)[0], 1)
return self.ix_to_char[ix[-1]]

Now I repeatedly call predict, character by character with a passage from the book:

affording a glancing bird's eye view of what has been promiscuously said, thought, fancied, and sung

This is, character by character, what the neural network predicted as coming next:

uu ,,seoillao in' iidyoslmt arerae..oe tuietiwet ,auymec,esnl y.aels weiddo, t teess atueietesis;o oenly'r o ,hole ietrnse aluunenisat,g,e tbelalivt e io ai "Colool,nooedqei,eo aernito,enyitoo

In other words, total garbage.

Am I doing something wrong, or are LSTMs in this fashion only capable of hallucinating text, and not predict what comes up next with reasonable accuracy based on training data?

• You didn't specify the key parameter - number of epochs. Also did you try network with only one LSTM layer? – Jakub Bartczuk Jan 27 '18 at 19:36
• @JakubBartczuk I've tested at a small number of epochs (~20) and now at ~2000 and both exhibit the same issue. I did not try the single layer. The behavior I'm seeing above is just really puzzling to me, where the network seemingly goes from well-behaved to total chaos. – orlp Jan 27 '18 at 20:35
• I haven't looked into the code too much, but perhaps you could take a look into teacher/professor forcing? Are you ensuring you pass on a larger and larger sequence/context each time, and that this context is used correctly by the model? – information_interchange Aug 14 '18 at 4:48