Per the universal approximation theorem, feed forward neural networks can approximate any function up to an arbitrary level of precision on the domain that they are trained on, given a sufficient amount of neurons and layers.
So what is it about RNN, LSTM, GRU that makes them better then FFNNets at certain tasks like NLP and seq2seq ?
Shouldn't FFNNets be able to learn anything that RNN/LSTM/GRU can? In the same way that an FFNet can approximate any function, shouldn't it be able to approximate any type of RNN as well?
(I am assuming here that we are applying different models to the same data set, otherwise comparisons and universal approximation properties would be moot)