11
$\begingroup$

I am trying to train a bidirectional LSTM to do a sequential text-tagging task (particularly, I want to do automatic punctuation).

I use letters as the building-blocks: I represent each input letter with a 50-dimensional embedding vector, fed into a single 100-dimensional hidden layer, fed into a 100-dimensional output layer, which is fed into an MLP.

For training, I have a corpus with about 18 miliion letters - there are about 70,000 sentences with about 250 letters in each sentence. (I have used DyNet on Python 3 on Ubuntu 16.04 system).

The main problem is that training is awfully slow : each iteration of training takes about half a day. Since training usually takes about 100 iterations, it means I will have to wait over a month to get reasonable results.

I asked some other people that do deep learning, and they told me "deep learning is slow, you have to get used to it". Still, waiting over a month for training seems horribly slow.

Are these times common for training LSTM models? If not, what am I doing wrong, and what can I do to speed up the training?

Improve this question
$\endgroup$
6
  • $\begingroup$ unrelated to your question but 50 dimensions for each input letter seems a bit much, seeing as a one-hot encoding would only require 26 letters. $\endgroup$
    – shimao
    Commented Sep 16, 2018 at 15:32
  • $\begingroup$ @shimao there are also numbers and symbols, and I treat upper and lower case letters as different (it might be useful in punctuation). But maybe it is worth trying a shorter representation anyway. $\endgroup$
    – Erel Segal-Halevi
    Commented Sep 16, 2018 at 15:34
  • $\begingroup$ by iteration, do you mean a single pass of backpropagation, or a single pass through your entire dataset? $\endgroup$
    – shimao
    Commented Sep 16, 2018 at 15:37
  • 1
    $\begingroup$ and do you have a GPU? $\endgroup$
    – shimao
    Commented Sep 16, 2018 at 15:38
  • $\begingroup$ @shimao By iteration I mean a single pass through the entire dataset. I have a GPU but I am not sure how it can help since, as far as I understand, the iterations must be done sequentially - I cannot just do 100 iterations in parallel, right? $\endgroup$
    – Erel Segal-Halevi
    Commented Sep 16, 2018 at 18:10

2 Answers 2

Reset to default
10
$\begingroup$

However much it pains me to say this, Deep learning is slow, get used to it.

There are some things you could do to speed up your training though:

  • What GPU are you using? A friend of mine was doing some research on LSTM's last year and training them on her NVIDIA GTX7?? GPU. Since this was going painfully slow, they tried to train the network on a more modern CPU, which actually led to a speed-up by a non trivial factor.

  • What framework are you using? While most frameworks are somewhat comparable, I have heard rumors (https://arxiv.org/pdf/1608.07249.pdf) that some frameworks are slower than others. It might be worthwhile to switch frameworks if you're going to be doing a lot of training.

  • Is it possible to train your network on your company/university hardware? Universities and research companies usually have some powerful hardware at their disposal. If this is not an option, maybe you can look into using some cloud-computing power.

All these solutions obviously assume your model itself is as optimal as it can be (In terms of training time and accuracy), which is also something you need to consider, but is outside of the scope of this answer.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Another possible advice is that there are faster implementations dedicated for GPUs like CuDNNLSTM in Keras. $\endgroup$
    – Tim
    Commented Sep 17, 2018 at 9:45
  • 1
    $\begingroup$ So a month of training is common for LSTM applications? $\endgroup$
    – Erel Segal-Halevi
    Commented Sep 19, 2018 at 20:52
  • $\begingroup$ If you're running it on a single system, it probably is. I don't know what happens when you run it on a cluster of devices though. (These guys used 6 NVIDIA Tesla k40's and it took them 40h to reach convergence: arxiv.org/abs/1708.05604) $\endgroup$
    – shmoo6000
    Commented Sep 19, 2018 at 21:02
5
$\begingroup$

If anyone else is struggling with this, I had the same issue - my LSTM layer was adding about 1 hour per epoch to the training time. I realised it was because in my IDE (PyCharm) I'd used the automatic import option when trying to use a method I hadn't yet manually imported at the top of my script. By default the automatic import statement was

from tensorflow.python.keras.layers import Bidirectional, LSTM

When I changed this to

from tensorflow.keras.layers import Bidirectional, LSTM

Training time per epoch dropped to just 4 minutes.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.