0
$\begingroup$

Per the universal approximation theorem, feed forward neural networks can approximate any function up to an arbitrary level of precision on the domain that they are trained on, given a sufficient amount of neurons and layers.

So what is it about RNN, LSTM, GRU that makes them better then FFNNets at certain tasks like NLP and seq2seq ?

Shouldn't FFNNets be able to learn anything that RNN/LSTM/GRU can? In the same way that an FFNet can approximate any function, shouldn't it be able to approximate any type of RNN as well?

(I am assuming here that we are applying different models to the same data set, otherwise comparisons and universal approximation properties would be moot)

$\endgroup$
  • $\begingroup$ so can polynomial regression $\endgroup$ – seanv507 May 11 at 6:19
  • 1
    $\begingroup$ "given a sufficient amount of neurons and layers" - you are forgetting the amount of training data. $\endgroup$ – Stephan Kolassa May 11 at 6:24
  • $\begingroup$ @StephanKolassa the amount of training data is implicitly the same here, otherwise no comparisons between the different classes of models would be valid. $\endgroup$ – Akaike's Children May 11 at 8:24
2
$\begingroup$

The universal approximation theorem asserts that there exsits a feed forward neural network to approximate a function. From the wiki page "however, it does not touch upon the algorithmic learnability of those parameters". In other words, although the universal approximation guarantees the existance of such a neural network, we cannot guarantee that there is a way to train a neural network to reach this performance.

There are situations where LSTM are better suited to learning the underlying function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.