I am trying to train a bidirectional LSTM to do a sequential text-tagging task (particularly, I want to do automatic punctuation).
I use letters as the building-blocks: I represent each input letter with a 50-dimensional embedding vector, fed into a single 100-dimensional hidden layer, fed into a 100-dimensional output layer, which is fed into an MLP.
For training, I have a corpus with about 18 miliion letters - there are about 70,000 sentences with about 250 letters in each sentence. (I have used DyNet on Python 3 on Ubuntu 16.04 system).
The main problem is that training is awfully slow : each iteration of training takes about half a day. Since training usually takes about 100 iterations, it means I will have to wait over a month to get reasonable results.
I asked some other people that do deep learning, and they told me "deep learning is slow, you have to get used to it". Still, waiting over a month for training seems horribly slow.
Are these times common for training LSTM models? If not, what am I doing wrong, and what can I do to speed up the training?