I was interested in this project, so I cloned it and trained it on Moby Dick, for this challenge. The goal is to predict the next character given the past ground-truth characters. Overfitting is not an issue, if the neural network remembers actual passages, the better.

After training the neural network, which has the following structure:

model = Sequential()
model.add(LSTM(500, input_shape=(None, VOCAB_SIZE), return_sequences=True))
model.add(LSTM(500, return_sequences=True))
model.compile(loss="categorical_crossentropy", optimizer="rmsprop")

If we now feed one character to the neural network, take whatever it predicts and feed everything it has produced so far back into repeatedly, it hallucinates reasonable looking text:

and cold, Sabayow, dreaming is not neck to comes to my fing that foolish a queer part of a harpoon from the sea, vessel in ite harpooneers, sharkless and round by all his occasional things. But no more of a nose, ether have inditably upon the one city of the Sperm Whale; which the land ships, grieving whales, are perhaps, at the barroo arp then, the transparent shoul I felt upon discreding a compass. Nor did the Captain Black and a ghatel story is absunding his chiefims.

Done as such:

def generate_text(model, length, vocab_size, ix_to_char):
    # starting with random character
    ix = [np.random.randint(vocab_size)]
    y_char = [ix_to_char[ix[-1]]]
    X = np.zeros((1, length, vocab_size))
    for i in range(length):
        # appending the last predicted character to sequence
        X[0, i, :][ix[-1]] = 1
        print(ix_to_char[ix[-1]], end="")
        ix = np.argmax(model.predict(X[:, :i+1, :])[0], 1)
    return ('').join(y_char)

So now I thought I could feed it a sequence of ground truth, and figure that it'd give reasonable estimates for what is coming up next:

class Predictor:
    def __init__(self, model, vocab_size, ix_to_char):
        self.model = model
        self.vocab_size = vocab_size
        self.ix_to_char = ix_to_char
        self.char_to_ix = {v:k for k, v in ix_to_char.items()}
        self.past = []

    def predict(self, last_c):
        X = np.zeros((1, len(self.past), self.vocab_size))
        for i in range(len(self.past)):
            X[0, i, self.past[i]] = 1.
        ix = np.argmax(self.model.predict(X)[0], 1)
        return self.ix_to_char[ix[-1]]

Now I repeatedly call predict, character by character with a passage from the book:

affording a glancing bird's eye view of what has been promiscuously said, thought, fancied, and sung

This is, character by character, what the neural network predicted as coming next:

uu ,,seoillao in' iidyoslmt arerae..oe tuietiwet ,auymec,esnl y.aels weiddo, t teess atueietesis;o oenly'r o ,hole ietrnse aluunenisat,g,e tbelalivt e io ai "Colool,nooedqei,eo aernito,enyitoo

In other words, total garbage.

Am I doing something wrong, or are LSTMs in this fashion only capable of hallucinating text, and not predict what comes up next with reasonable accuracy based on training data?

  • 1
    $\begingroup$ You didn't specify the key parameter - number of epochs. Also did you try network with only one LSTM layer? $\endgroup$ – Jakub Bartczuk Jan 27 '18 at 19:36
  • $\begingroup$ @JakubBartczuk I've tested at a small number of epochs (~20) and now at ~2000 and both exhibit the same issue. I did not try the single layer. The behavior I'm seeing above is just really puzzling to me, where the network seemingly goes from well-behaved to total chaos. $\endgroup$ – orlp Jan 27 '18 at 20:35
  • $\begingroup$ I haven't looked into the code too much, but perhaps you could take a look into teacher/professor forcing? Are you ensuring you pass on a larger and larger sequence/context each time, and that this context is used correctly by the model? $\endgroup$ – information_interchange Aug 14 '18 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.