batch-training LSTM with pretrained & out-of-vocabulary word embeddings in keras

Question

My goal is to batch-train an RNN LSTM mode using Stochastic Gradient Descent to predict named entities from labeled text in keras. The input to my model are word-sized units. I chose to represent words using pretrained GloVe embeddings.

Since my training data comes from a specialized domain, it contains words that do not have matches in GloVe (out-of-vocabulary words). For these I generated pseudo-random embeddings by sampling from a truncated normal distribution based on summary statistics from GloVe embeddings; I also made sure that tokens of the same type have the same embedding.

Given this heterogenous dataset, the usual approach of creating a lookup table of embeddings equal to the length of the vocabulary and generating embedding batches on the fly based on the lookup table was impractical. So instead, I padded the sequences to MAX_SEQ_LEN and pre-generated my batches.

So my batch data is a NUM_BATCHES-long list of (X,y) tuples where:

• X are inputs consisting of embedding matrices of shape MAX_SEQ_LEN, D], where D is the dimensionality of the embedding;
• y are labels consisting of vectors of length MAX_SEQ_LEN consiting of integers of ranging from 0 (padded cell response) to NUM_CLASSES.

My question:

I am using an external loop for whole data epochs and then an internal queue within the keras model.fit() fuction to feed my batches one by one. This does not seem like the most appropriate way to do this. I wonder because my learning curve (see loss function plot) is quite jagged, even after decreasing the learning rate; I’d expected batch-training to yield smoother learning curves. What is a more appropriate way to implement batch training given my mixed embedding and fixed batch constraints?

from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed, Bidirectional
from keras import optimizers

# define LSTM
model = Sequential()
model.add(Bidirectional(LSTM(HIDDEN_SIZE, return_sequences=True), input_shape=(MAX_SEQ_LEN, D)))
model.add(TimeDistributed(Dense(N_CLASSES, activation='sigmoid')))
model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizers.SGD(lr=.1), metrics=['mse'])

# train LSTM
for epoch in range(1):

   for X,y in train:

        # reshape input and output data to be suitable for LSTMs
        X = X.reshape(1, MAX_SEQ_LEN, D)
        y = y.reshape(1, MAX_SEQ_LEN, 1)

        # fit model for one epoch on this sequence
        model.fit(X, y, epochs=1, batch_size=1, verbose=2)

Answer 1

If the batch size is small and the size of the dataset is not massive, you are likely to get jagged curves, as the gradient of the loss could be noisy. Note that noisy gradients are what makes mini-batching effective: noise helps escape local minima. That said, the plots you are showing refer to a single epoch, is it correct? If so, they are showing you how the loss behaves in a single epoch, for each data batch. During a single epoch, you process potentially different batches, so the loss might oscillate. Instead, you should see the learning curve where you plot different epochs on the x-axis and the average loss over all batches in the y-axis.