I'm currently working with LSTMs and BiLSTMs, using Keras as library (TF backend). Following Tutorials and reading some papers, I found out that the sequences used are mostly quite short.

What I do have on the other hand, are really long pairs with a sentence and a related document (or more than one). For each pair of I have a binary label. I want to learn, if the sentence makes an assumption about the document.

Since the documents are quite long, I checked how many documents I've got on average (around 4) for a related sentence and how long these docs usually are. Based on this I found out, that the trade-off between sequence length for the documents and loosing as few as possible information, is at around 2500 tokens. For the sentence-part I have a length of 100 tokens at max. If I have more than one document, I use 2500/#docs tokens for each document and concatenate them. So I have sequences of 2600 tokens for each sample.

I then create two BiLSTMs, one for the sentence, one for the doc(s) and concatenate their result. I use GloVe embeddings (100d, 400k length). I pad and mask the sequences. I make the embedding trainable, since otherwise there is almost no change in accuracy over the epochs (maybe I need to run the model for a lot more epochs, which I just can't do).

Here is my problem: It takes all the memory on the GPU for a batch_size of 50 already. To run 3400 samples it takes 12min for a single epoch. This makes playing around with the hyperparameters and overall testing quite impossible. I shortened the length of the sequences by the factor 2 (to 1300 tokens), increased batch size to 150, but still it takes a bit more than 3 mins each epoch. I noticed the results get a bit worse, but not much, though I can't really compare the results of higher epochs because of the computation time.

I have 4 questions about this problem:

  1. What is the usual sequence length you use for LSTMs/BiLSTMs? Are my sequences just too long? I could split up pairs into many (shorter) pairs, but there is a relation between the documents which would be lost (I guess) and this would increase the samples of course so it might not make a difference in computation time.
  2. I'm already filtering out stopwords and due to the GloVe embeddings a lot of words get just zero vectors and masked out (though this might not decrease the computation time(?)). What other options do I have for my problem in order to lose as few information as possible? Do you have any suggestions for me?
  3. I get the best results if I stop after only 2 or 3 epochs. The results each epoch vary a lot, but they are almost always best (i.e. for each fold in my 10 fold cross-validation) for the first 2-5 epochs. Then they usually drop very fast. Most probably because I start to overfit on the training data. Still, at this point my model seems to be quite unstable, because of the variations in accuracy/f1. Does this mean I can't really make reliable predictions after that few epochs? I see a lot of examples for LSTMs running for hundreds and thousands of epochs.
  4. The data is quite unblanced (almost 80% for one class). I calculate class weights and use them for training, so the bigger class gets penalized. Does the accuracy output (see below) of 1.0 after only 6 epochs actually mean I perfectly predict the training labels and I overfitted already? It seems odd, that it overfits that fast with that many parameters to learn. (Because of the unbalanced data, I output f1 macro on the validation data after each epoch). I already added a dropout of 0.2 after each BiLSTM.

(output of one fold showing 6 epochs and possible overfitting)

Epoch1: 3405/3405 [==============================] - 184s - loss: 0.6863 - acc: 0.6767 - val_loss: 0.6969 - val_acc: 0.4670 - f1 (macro): 0.453781392694
Epoch2: 3405/3405 [==============================] - 168s - loss: 0.3600 - acc: 0.8394 - val_loss: 0.5692 - val_acc: 0.7625 - f1 (macro): 0.633288896534
Epoch3: 3405/3405 [==============================] - 172s - loss: 0.0423 - acc: 0.9897 - val_loss: 0.7367 - val_acc: 0.7704 - f1 (macro): 0.612433442646
Epoch4: 3405/3405 [==============================] - 173s - loss: 0.0094 - acc: 0.9982 - val_loss: 0.8715 - val_acc: 0.7757 - f1 (macro): 0.582041335288
Epoch5: 3405/3405 [==============================] - 174s - loss: 0.0020 - acc: 0.9997 - val_loss: 0.8572 - val_acc: 0.7836 - f1 (macro): 0.599469017424
Epoch6: 3405/3405 [==============================] - 175s - loss: 9.6727e-04 - acc: 1.0000 - val_loss: 0.9114 - val_acc: 0.7810 - f1 (macro): 0.586315277285
Epoch7: 3405/3405 [==============================] - 173s - loss: 8.9306e-04 - acc: 1.0000 - val_loss: 0.9645 - val_acc: 0.7863 - f1 (macro): 0.590663626545

I'm thankful for any advise!


1 Answer 1

  1. 250-500 timesteps is the 'norm' if you are not using any tricks to retain memory over very long sequences.

  2. No.

  3. Early stopping is a thing, but your problem seems to be in... how you've formulated your problem and posed it to your network.

  4. 20% target class is likely not an issue.

The way you've posed your problem seems weird, and it also seems like you're trying to something very high level (identifying assumptions) with a single-layer bilstm, which may not be appropriate. Perfect accuracy from a short training time generally means there is some sort of a problem in how the question is being posed or how the data are structured (e.g., leakage). How does your model perform on unseen documents?

I would think through what it is you're trying to force your network to do, and if that is even appropriate. I would also look into document summarization (e.g., doc2vec and using a normal NN, or reducing the document to a sequence of sentence-level meaning vectors, rather than word-level).

To reduce your documents to sequences of word vectors, you train a standard encoder-decoder to reconstruct sentences. You feed it your word vectors, then have the decoder reconstruct those words. Once it's well-trained, you can feed it all your training data (all sentences) and extract the final encoder hidden states for each sentence. Then, each document is a far far shorter sequence of X-length 'thought vectors'. You may also use an approach called 'skip-thought'.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.