Is LSTM (Long Short-Term Memory) dead?

Question

From my own experience, LSTM has a long training time, and does not improve performance significantly in many real world tasks.

To make the question more specific, I want to ask when LSTM will work better than other deep NN (may be with real world examples)? I know LSTM captures the sequential relationship in data, but is it really necessary?

Most demos on related topic are meaningless. They just focus on toy data e.g., IMDB review, where simple logistic regression will get very good results. I do not see any value of using LSTM which has huge computational cost but marginal improvements (if there are any).

Even with these toy examples, I did not find any good use cases that LSTM can solve very well but other models cannot.

Answer 1

Maybe. But RNNs aren't.

Transformers learn "pseudo-temporal" relationships; they lack the true recurrent gradient that RNNs have, and thus extract fundamentally different features. This paper, for example, shows that the standard transformers are difficult to optimize in reinforcement learning settings, especially in memory-intensive environments. They do, however, eventually design a variant surpassing LSTMs.

Where are RNNs still needed?

Long memory tasks. Very long memory. IndRNNs have show ability to remember for 5000 timesteps, where LSTM barely manages 1000. A transformer is quadratic in time-complexity whereas RNNs are linear, meaning good luck processing even a single iteration of 5000 timesteps. If that isn't enough, the recent Legendre Memory Units have demonstrated memory of up to 512,000,000 timesteps; I'm unsure the world's top supercomputer could fit the resultant 1E18 tensor in memory.

Aside reinforcement learning, signal applications are memory-demanding - e.g. speech synthesis, video synthesis, seizure classification. While CNNs have shown much success on these tasks, many utilize RNNs inserted in later layers; CNNs learn spatial features, RNNs temporal/recurrrent. An impressive 2019 paper's network manages to clone a speaker's voice from a only a 5 second sample, and it uses CNNs + LSTMs.

Memory vs. Feature Quality:

One doesn't warrant the other; "quality" refers to information utility for a given task. For sentences with 50 words, for example, model A may classify superior to model B, but fail dramatically with 100 where B would have no trouble. This exact phenomenon is illustrated in the recent Bistable Recurrent Cell paper, where the cell shows better memory for longer sequences, but is outdone by LSTMs on shorter sequences. An intuition is, LSTMs' four-gated networking permits for greater control over information routing, and thus richer feature extraction.

Future of LSTMs?

My likeliest bet is, some form of enhancement - like a Bistable Recurrent Cell, maybe with attention, and recurrent normalization (e.g. LayerNorm or Recurrent BatchNorm). BRC's design is based on control theory, and so are LMUs; such architectures enjoy self-regularization, and there's much room for further innovation. Ultimately, RNNs cannot be "replaced" by non-recurrent architectures, and will thus perform superior on some tasks that demand explicitly recurrent features.

Recurrent Transformers

If we can't do away with recurrence, can't we just incorporate it with transformers somehow? Yes: Universal Transformers. Not only is there recurrence, but variable input sequences are supported, just like in RNNs. Authors go so far as to argue that UTs are Turing complete; whether that's true I haven't verified, but even if it is, it doesn't warrant practical ability to fully harness this capability.

Bonus: It helps to visualize RNNs to better understand and debug them; you can see their weights, gradients, and activations in action with See RNN, a package of mine (pretty pics included).

---

Update 6/29/2020: new paper redesigns transformers to operate in time dimension with linear, O(N), complexity: Transformers are RNNs. Mind the title though; from section 3.4: "we consider recurrence with respect to time and not depth". So they are a kind of RNN, but still differ from 'traditional' ones. I've yet to read it, seems promising; a nice video explanation here.

Answer 2

It is funny that you ask now, since just today I came across a paper by Wang, Khabsa, and Ma (2020) To Pretrain or Not to Pretrain who show that if you have large enough training set, the difference in performance between huge, "SOTA" model (RoBERTa), and LSTMs is small for NLP task. There was another recent paper by Merity (2019) Single Headed Attention RNN showing similar results, the abstract is worth quoting in full

The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon. We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result. We take a previously strong language model based only on boring LSTMs and get it to within a stone's throw of a stone's throw of state-of-the-art byte level language model results on enwik8. This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts with minimal computation. Take that Sesame Street.

I don't think there's much to add.

Here is another example from very recent paper by Abnar, Dehghani, and Zuidema (2020) Transferring Inductive Biases through Knowledge Distillation

Several studies, however, have shown that LSTMs can perform better than Transformers on tasks requiring sensitivity to (linguistic) structure, especially when the data is limited [37, 6]. This is mainly due to the recurrent inductive biases of LSTMs that helps them better model the hierarchical structure of the inputs.

hence authors show how distilling information from LSTMs can positively impact Transformer model. This another, of many, examples that LSTMs, and RNNs in general, are used and perform good for a particular class of problems. Sure, they have limitations, but for language they are standard model, that is taught on on every NLP course (like Stanford's CS224n), and mentioned in every modern handbook on this topic. The above examples focus on language data, because in this area this model is very popular, but of course it is successfully applied to other kinds of time-series data as well, as mentioned in other answers.

Answer 3

Our group recently built an LSTM model in a real world application. At first we had used other approaches, but then we decided to include features that were measurements taken over time, but of variable length - so for one person, we would have 15 measurements (of the same parameter) taken over a 3-month period, for another we would have 20 measurements over a 2-month period, and so on. Other features were present once per person, e.g. gender.

In this situation, standard time series approaches turned out to be unusable, since they expected us to have an equal number of measurements per person, taken at equal intervals. LSTM allowed us to build a model predicting if a certain event will occur for a person, using the variable length measurements combined with the once-per-person measurements.

We also compared our model to a simpler regression model using only one value per time-varying parameter (I forget what it was, probably the average value over time) and to a regression model using three measurements per time-varying feature per person and treating them as measurements of independent variables. The LSTM model had much better accuracy than both of these models, especially for the class of persons for whom the event occurred.

I know that this is just one counterexample, and LSTM is not the only algorithm to deal with that kind of situation - but the way your question is stated lends itself to counterexamples, and statistics/ML would be an impoverished area if we didn't have different tools to choose from.

Answer 4

LSTM is a statistical method. It is not alive so it cannot be dead. It can be useful though. Any statistical method is another tool in a box. If one does not work it is good to have an alternative.

LSTM is good for language recognition tasks where context is important. It is also good for forecasting time series. The M4 competition was won by LSTM.

If it was not useful there would not be a significant body of research dedicated to it. However as far as I know there is no proof that LSTM is inferior to any other method in some meaningful sense, i.e. the class of problems which LSTM is able to solve is smaller than logistic regression, etc.