Given the universal approximation theorem, why are LSTM better than feed forward neural networks at certain tasks?

Question

Per the universal approximation theorem, feed forward neural networks can approximate any function up to an arbitrary level of precision on the domain that they are trained on, given a sufficient amount of neurons and layers.

So what is it about RNN, LSTM, GRU that makes them better then FFNNets at certain tasks like NLP and seq2seq ?

Shouldn't FFNNets be able to learn anything that RNN/LSTM/GRU can? In the same way that an FFNet can approximate any function, shouldn't it be able to approximate any type of RNN as well?

(I am assuming here that we are applying different models to the same data set, otherwise comparisons and universal approximation properties would be moot)

Answer 1

The universal approximation theorem asserts that there exsits a feed forward neural network to approximate a function. From the wiki page "however, it does not touch upon the algorithmic learnability of those parameters". In other words, although the universal approximation guarantees the existance of such a neural network, we cannot guarantee that there is a way to train a neural network to reach this performance.

There are situations where LSTM are better suited to learning the underlying function.