# batch-training LSTM with pretrained & out-of-vocabulary word embeddings in keras

My goal is to batch-train an RNN LSTM mode using Stochastic Gradient Descent to predict named entities from labeled text in keras. The input to my model are word-sized units. I chose to represent words using pretrained GloVe embeddings.

Since my training data comes from a specialized domain, it contains words that do not have matches in GloVe (out-of-vocabulary words). For these I generated pseudo-random embeddings by sampling from a truncated normal distribution based on summary statistics from GloVe embeddings; I also made sure that tokens of the same type have the same embedding.

Given this heterogenous dataset, the usual approach of creating a lookup table of embeddings equal to the length of the vocabulary and generating embedding batches on the fly based on the lookup table was impractical. So instead, I padded the sequences to MAX_SEQ_LEN and pre-generated my batches.

So my batch data is a NUM_BATCHES-long list of (X,y) tuples where:

• X are inputs consisting of embedding matrices of shape MAX_SEQ_LEN, D], where D is the dimensionality of the embedding;
• y are labels consisting of vectors of length MAX_SEQ_LEN consiting of integers of ranging from 0 (padded cell response) to NUM_CLASSES.

My question:

I am using an external loop for whole data epochs and then an internal queue within the keras model.fit() fuction to feed my batches one by one. This does not seem like the most appropriate way to do this. I wonder because my learning curve (see loss function plot) is quite jagged, even after decreasing the learning rate; I’d expected batch-training to yield smoother learning curves. What is a more appropriate way to implement batch training given my mixed embedding and fixed batch constraints?

from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed, Bidirectional
from keras import optimizers

# define LSTM
model = Sequential()
model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizers.SGD(lr=.1), metrics=['mse'])

# train LSTM
for epoch in range(1):

for X,y in train:

# reshape input and output data to be suitable for LSTMs
X = X.reshape(1, MAX_SEQ_LEN, D)
y = y.reshape(1, MAX_SEQ_LEN, 1)

# fit model for one epoch on this sequence
model.fit(X, y, epochs=1, batch_size=1, verbose=2)