Maybe. But RNNs aren't.
Transformers learn "pseudo-temporal" relationships; they lack the true recurrent gradient that RNNs have, and thus extract fundamentally different features. This paper, for example, shows that the standard transformers are difficult to optimize in reinforcement learning settings, especially in memory-intensive environments. They do, however, eventually design a variant surpassing LSTMs.
Where are RNNs still needed?
Long memory tasks. Very long memory. IndRNNs have show ability to remember for 5000 timesteps, where LSTM barely manages 1000. A transformer is quadratic in time-complexity whereas RNNs are linear, meaning good luck processing even a single iteration of 5000 timesteps. If that isn't enough, the recent Legendre Memory Units have demonstrated memory of up to 512,000,000 timesteps; I'm unsure the world's top supercomputer could fit the resultant 1E18 tensor in memory.
Aside reinforcement learning, signal applications are memory-demanding - e.g. speech synthesis, video synthesis, seizure classification. While CNNs have shown much success on these tasks, many utilize RNNs inserted in later layers; CNNs learn spatial features, RNNs temporal/recurrrent. An impressive 2019 paper's network manages to clone a speaker's voice from a only a 5 second sample, and it uses CNNs + LSTMs.
Memory vs. Feature Quality:
One doesn't warrant the other; "quality" refers to information utility for a given task. For sentences with 50 words, for example, model A may classify superior to model B, but fail dramatically with 100 where B would have no trouble. This exact phenomenon is illustrated in the recent Bistable Recurrent Cell paper, where the cell shows better memory for longer sequences, but is outdone by LSTMs on shorter sequences. An intuition is, LSTMs' four-gated networking permits for greater control over information routing, and thus richer feature extraction.
Future of LSTMs?
My likeliest bet is, some form of enhancement - like a Bistable Recurrent Cell, maybe with attention, and recurrent normalization (e.g. LayerNorm or Recurrent BatchNorm). BRC's design is based on control theory, and so are LMUs; such architectures enjoy self-regularization, and there's much room for further innovation. Ultimately, RNNs cannot be "replaced" by non-recurrent architectures, and will thus perform superior on some tasks that demand explicitly recurrent features.
If we can't do away with recurrence, can't we just incorporate it with transformers somehow? Yes: Universal Transformers. Not only is there recurrence, but variable input sequences are supported, just like in RNNs. Authors go so far as to argue that UTs are Turing complete; whether that's true I haven't verified, but even if it is, it doesn't warrant practical ability to fully harness this capability.
Bonus: It helps to visualize RNNs to better understand and debug them; you can see their weights, gradients, and activations in action with See RNN, a package of mine (pretty pics included).
Update 6/29/2020: new paper redesigns transformers to operate in time dimension with linear, O(N), complexity: Transformers are RNNs. Mind the title though; from section 3.4: "we consider recurrence with respect to time and not depth". So they are a kind of RNN, but still differ from 'traditional' ones. I've yet to read it, seems promising; a nice video explanation here.