I want to try to reproduce the result of the following article.
Rather than inputting words into the RNN you simply input a byte as a 256 long vector. At the output you get back another 256 length vector which should represent the next byte. I am using Keras with the theano backend. I have run this model with a few different variations for a few hours now. The best accuracy I can achieve is about 30%. My program converges on that 30% real fast and stops improving. My loss function also remains the same. When I run the test and for instance start the first character with 'T' it outputs 'The _________________________' followed by a bunch of spaces. This makes sense since The is probably the most likely word and there are a lot of spaces in the document (A 5MB text file containing the complete works of Shakespeare).
As for my model I am using the LTSM layers each with 128 outputs. I have tried using one two and three layers. I have also tried different output sizes. Nothing makes a difference. It always stagnates at 30%. I am also not really sure I am doing it correctly. I went of some examples in the Keras documentation. I am only using one time step since I set the stateful flag in the LSTM so that it will connect each batch to the next. Is this ok or do I need to add time steps? Also do you see anything else wrong with it?
import random import numpy as np batch_size = 32 #byte to vector def btov(b): v = np.zeros(256) v[int(b)] = 1 return v #vector to byte def vtob(v): return np.argmax(v.flatten()) data = open('/home/chase/Desktop/shakespeare.txt', 'rb').read() print('Bytes of Training Data:', len(data)) from keras.models import * from keras.layers import * from keras.optimizers import * from keras.callbacks import * model = Sequential() model.add(LSTM(128, batch_input_shape = (batch_size, 1, 256), stateful = True, return_sequences = True)) model.add(LSTM(128, stateful = True)) model.add(Dropout(0.1)) model.add(Dense(256, activation = 'softmax')) model.compile(optimizer = Adam(), loss = 'categorical_crossentropy', metrics = ['accuracy']) model.summary() #create batched training data def gen_training(): x_train = np.zeros((batch_size, 1, 256)) y_train = np.zeros((batch_size, 256)) #each batch has a random starting point in the data ix = [random.randint(0, len(data)) for _ in range(batch_size)] while True: #for every batch for b in range(batch_size): #load the byte at ix into x x_train[b] = btov(data[ix[b]]) #advance to the next byte ix[b] += 1 ix[b] %= len(data) #the next byte is the desired output y_train[b] = btov(data[ix[b]]) yield (x_train, y_train) model.fit_generator(gen_training(), samples_per_epoch = batch_size * 10000, nb_epoch = 1000, callbacks = [ModelCheckpoint('best_loss.h5', monitor = 'loss', verbose = 0, save_best_only = True, mode = 'auto'), ModelCheckpoint('best_acc.h5', monitor = 'acc', verbose = 0, save_best_only = True, mode = 'auto'), ModelCheckpoint('last.h5', monitor = 'loss', verbose = 0, save_best_only = False, mode = 'auto')]) ''' This is my test of the model. When I run this I would load the weights and comment out the fitting. First I get some data in the memory with the seed. Then I feed an initial character into the model such as T in this case. I read the output of the predicting and feed that back in on the next loop. When I have T for example it prints [The ] followed by a bunch of spaces. ''' seed = 'I grant thou wert not married to my muse\nAnd therefore mayst without attaint o\'erlook' for c in seed: x_prime = np.zeros((batch_size, 1, 256)) x_prime = btov(ord(c)) model.predict(x_prime, batch_size = batch_size) x = np.zeros(256) x[ord('T')] = 1 s = '' for i in range(100): s += chr(vtob(x)) x_prime = np.zeros((batch_size, 1, 256)) x_prime = x x = model.predict(x_prime, batch_size = batch_size).flatten() print(x[ord(' ')]) x = btov(vtob(x)) print('[', s, ']')