Deep learning with a lot of training data I am building a bidirectional LSTM to do a sequential text-tagging task (particularly, automatic punctuation). Usually, the training is done in iterations, where in each iteration, the entire training data is read, the loss is calculated, and the weights are updated to decrease the loss. But, I have a lot of training data; with even 1% of my training data, an iteration takes half a day. Moreover, I keep getting new training data, so even if I run an iteration on current training data, by the time it ends I will have more.
Obviously I can ignore or sub-sample the training data to make the iterations faster, but this seems like a wasteful way to train, since it does not use all available data. Is there a way to train the LSTM, that takes advantage of a large amount, possibly unbounded, of training data?
Ideally, I would like to create a classifier that improves with time - when I bring it new samples, it improves its performance, without having to run on previous samples. Is this possible with LSTM?
Also asked here: https://ai.stackexchange.com/q/8018/8684 with no answers yet
 A: The standard approach would be to have a generator that keeps on generating training batches (some subset of the total data such as 5000 records at once) on the fly, which would not really care, if the database keeps growing in the background. You can of course speed things up with more hardware (e.g. with the MPI version of TensorFlow you should be able to use multiple severs with multiple GPUs each, if you have access to such a system/the budget to rent it).
I would guess that your main concerns when doing this would be that firstly a model without some regularization may overfit, if you keep iterating. That might be addressed by some suitable regularization (e.g. drop-out) and having lots of data (seems to potentially be the case here). Additionally, the underlying process might change over time. You may wish to keep an eye on whether you do better training on all the data, doing iterations only on records within some training time window (e.g. last x months) that keeps moving over time, or with every now and then re-training from scratch on the most recent data within some window. It is not certain which of these would work best and it may be best to empirically evaluate it.
