Does my recurrent neural network look correct?

Question

I want to try to reproduce the result of the following article.

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Rather than inputting words into the RNN you simply input a byte as a 256 long vector. At the output you get back another 256 length vector which should represent the next byte. I am using Keras with the theano backend. I have run this model with a few different variations for a few hours now. The best accuracy I can achieve is about 30%. My program converges on that 30% real fast and stops improving. My loss function also remains the same. When I run the test and for instance start the first character with 'T' it outputs 'The _________________________' followed by a bunch of spaces. This makes sense since The is probably the most likely word and there are a lot of spaces in the document (A 5MB text file containing the complete works of Shakespeare).

As for my model I am using the LTSM layers each with 128 outputs. I have tried using one two and three layers. I have also tried different output sizes. Nothing makes a difference. It always stagnates at 30%. I am also not really sure I am doing it correctly. I went of some examples in the Keras documentation. I am only using one time step since I set the stateful flag in the LSTM so that it will connect each batch to the next. Is this ok or do I need to add time steps? Also do you see anything else wrong with it?

import random
import numpy as np

batch_size = 32

#byte to vector
def btov(b):
    v = np.zeros(256)
    v[int(b)] = 1
    return v

#vector to byte
def vtob(v):
    return np.argmax(v.flatten())

data = open('/home/chase/Desktop/shakespeare.txt', 'rb').read()
print('Bytes of Training Data:', len(data))

from keras.models import *
from keras.layers import *
from keras.optimizers import *
from keras.callbacks import *

model = Sequential()
model.add(LSTM(128, batch_input_shape = (batch_size, 1, 256), stateful = True, return_sequences = True))
model.add(LSTM(128, stateful = True))
model.add(Dropout(0.1))
model.add(Dense(256, activation = 'softmax'))

model.compile(optimizer = Adam(),
              loss = 'categorical_crossentropy', 
              metrics = ['accuracy'])

model.summary()

#create batched training data
def gen_training():
    x_train = np.zeros((batch_size, 1, 256))
    y_train = np.zeros((batch_size, 256))

    #each batch has a random starting point in the data
    ix = [random.randint(0, len(data)) for _ in range(batch_size)]
    while True:
        #for every batch
        for b in range(batch_size):
            #load the byte at ix into x
            x_train[b][0] = btov(data[ix[b]])

            #advance to the next byte
            ix[b] += 1
            ix[b] %= len(data)

            #the next byte is the desired output
            y_train[b] = btov(data[ix[b]])

        yield (x_train, y_train)


model.fit_generator(gen_training(),
                    samples_per_epoch = batch_size * 10000,
                    nb_epoch = 1000,
                    callbacks = [ModelCheckpoint('best_loss.h5', monitor = 'loss', verbose = 0, save_best_only = True, mode = 'auto'),
                                 ModelCheckpoint('best_acc.h5', monitor = 'acc', verbose = 0, save_best_only = True, mode = 'auto'),
                                 ModelCheckpoint('last.h5', monitor = 'loss', verbose = 0, save_best_only = False, mode = 'auto')])

'''
This is my test of the model.  When I run this I would load the weights and comment out
the fitting.  First I get some data in the memory with the seed.  Then I feed an initial character
into the model such as T in this case.  I read the output of the predicting and feed that back
in on the next loop.  When I have T for example it prints [The                           ]
followed by a bunch of spaces.
'''

seed = 'I grant thou wert not married to my muse\nAnd therefore mayst without attaint o\'erlook'
for c in seed:
    x_prime = np.zeros((batch_size, 1, 256))
    x_prime[0][0] = btov(ord(c))
    model.predict(x_prime, batch_size = batch_size)

x = np.zeros(256)
x[ord('T')] = 1  
s = ''
for i in range(100):
    s += chr(vtob(x))
    x_prime = np.zeros((batch_size, 1, 256))
    x_prime[0][0] = x
    x = model.predict(x_prime, batch_size = batch_size)[0].flatten()
    print(x[ord(' ')])
    x = btov(vtob(x))

print('[', s, ']')

Answer 1

Well I spent all day figuring it out. First off I was way under-fitting the data. Now I have 3 LSTM layers of size 1024. Also I needed to use a sampling function on the output. Now my model looks like this.

model = Sequential()
model.add(LSTM(1024, batch_input_shape = (batch_size, 1, len(bytes_set)), stateful = True, return_sequences = True))
model.add(Dropout(0.3))
model.add(LSTM(1024, stateful = True, return_sequences = True))
model.add(Dropout(0.3))
model.add(LSTM(1024, stateful = True, return_sequences = False))
model.add(Dropout(0.3))
model.add(Dense(len(bytes_set), activation = 'softmax'))

In addition I found the following sampling function online which makes the results much better rather than all spaces.

def sample(a, temperature = 1.0):
    a = np.log(a) / temperature
    a = np.exp(a) / np.sum(np.exp(a))
    return np.argmax(np.random.multinomial(1, a, 1))

Now after only running 30,000 iterations with a batch size of 64 characters I get output which looks like this.

If the more for the conbe mance of plainesss rishrrith?
    And of which be have many and frief of all not the pake and so,
    Marrcoing him him the gaest, that out to live him, in seal onier this for that man of a gright my do fare thy warrm of him here the arest distan the teer for boours of this take in he speak,
    The laws his our fear friend, and sur that I fay the cance was whe say may such despratures whe cord whave you dee you the lend is the frain of a cear he ronour?
    On she tatch foulow of the right of me the for so reter greats so him the sur was a dot.
    Seee the more from thou she of su the flie sweet the gormpre whelefive mad of the the or a foul; and this deear canndse have shat grose is de hear lay me,
    The reI where that the day the would like to frait, die hearts thing the meent,

Not very coherent but my network defiantly learned old English grammar very quickly. I am actually surprised at how fast. The text file I used for training is only 5MB but my model has 21 million parameters. I feel like this would cause overfitting.