I'm currently working with LSTMs and BiLSTMs, using Keras as library (TF backend). Following Tutorials and reading some papers, I found out that the sequences used are mostly quite short.
What I do have on the other hand, are really long pairs with a sentence and a related document (or more than one). For each pair of I have a binary label. I want to learn, if the sentence makes an assumption about the document.
Since the documents are quite long, I checked how many documents I've got on average (around 4) for a related sentence and how long these docs usually are. Based on this I found out, that the trade-off between sequence length for the documents and loosing as few as possible information, is at around 2500 tokens. For the sentence-part I have a length of 100 tokens at max. If I have more than one document, I use 2500/#docs tokens for each document and concatenate them. So I have sequences of 2600 tokens for each sample.
I then create two BiLSTMs, one for the sentence, one for the doc(s) and concatenate their result. I use GloVe embeddings (100d, 400k length). I pad and mask the sequences. I make the embedding trainable, since otherwise there is almost no change in accuracy over the epochs (maybe I need to run the model for a lot more epochs, which I just can't do).
Here is my problem: It takes all the memory on the GPU for a batch_size of 50 already. To run 3400 samples it takes 12min for a single epoch. This makes playing around with the hyperparameters and overall testing quite impossible. I shortened the length of the sequences by the factor 2 (to 1300 tokens), increased batch size to 150, but still it takes a bit more than 3 mins each epoch. I noticed the results get a bit worse, but not much, though I can't really compare the results of higher epochs because of the computation time.
I have 4 questions about this problem:
- What is the usual sequence length you use for LSTMs/BiLSTMs? Are my sequences just too long? I could split up pairs into many (shorter) pairs, but there is a relation between the documents which would be lost (I guess) and this would increase the samples of course so it might not make a difference in computation time.
- I'm already filtering out stopwords and due to the GloVe embeddings a lot of words get just zero vectors and masked out (though this might not decrease the computation time(?)). What other options do I have for my problem in order to lose as few information as possible? Do you have any suggestions for me?
- I get the best results if I stop after only 2 or 3 epochs. The results each epoch vary a lot, but they are almost always best (i.e. for each fold in my 10 fold cross-validation) for the first 2-5 epochs. Then they usually drop very fast. Most probably because I start to overfit on the training data. Still, at this point my model seems to be quite unstable, because of the variations in accuracy/f1. Does this mean I can't really make reliable predictions after that few epochs? I see a lot of examples for LSTMs running for hundreds and thousands of epochs.
- The data is quite unblanced (almost 80% for one class). I calculate class weights and use them for training, so the bigger class gets penalized. Does the accuracy output (see below) of 1.0 after only 6 epochs actually mean I perfectly predict the training labels and I overfitted already? It seems odd, that it overfits that fast with that many parameters to learn. (Because of the unbalanced data, I output f1 macro on the validation data after each epoch). I already added a dropout of 0.2 after each BiLSTM.
(output of one fold showing 6 epochs and possible overfitting)
Epoch1: 3405/3405 [==============================] - 184s - loss: 0.6863 - acc: 0.6767 - val_loss: 0.6969 - val_acc: 0.4670 - f1 (macro): 0.453781392694 Epoch2: 3405/3405 [==============================] - 168s - loss: 0.3600 - acc: 0.8394 - val_loss: 0.5692 - val_acc: 0.7625 - f1 (macro): 0.633288896534 Epoch3: 3405/3405 [==============================] - 172s - loss: 0.0423 - acc: 0.9897 - val_loss: 0.7367 - val_acc: 0.7704 - f1 (macro): 0.612433442646 Epoch4: 3405/3405 [==============================] - 173s - loss: 0.0094 - acc: 0.9982 - val_loss: 0.8715 - val_acc: 0.7757 - f1 (macro): 0.582041335288 Epoch5: 3405/3405 [==============================] - 174s - loss: 0.0020 - acc: 0.9997 - val_loss: 0.8572 - val_acc: 0.7836 - f1 (macro): 0.599469017424 Epoch6: 3405/3405 [==============================] - 175s - loss: 9.6727e-04 - acc: 1.0000 - val_loss: 0.9114 - val_acc: 0.7810 - f1 (macro): 0.586315277285 Epoch7: 3405/3405 [==============================] - 173s - loss: 8.9306e-04 - acc: 1.0000 - val_loss: 0.9645 - val_acc: 0.7863 - f1 (macro): 0.590663626545
I'm thankful for any advise!