1
$\begingroup$

In this paper by Bengio on page 7 he used the attractor network to analyse recurrent network. And in this paper it says that attractor network dominates the field of memory control. What's more recurrent network features its memory ability compared to previous ones. so I'm wondering what kind of role does it play in the field of recurrent neural network or neural network nowadays? I raise this question because 1) I have asked some experts who told me they had never met this before and 2) in the popular deeplearning textbook by Ian Goodfellow, Yoshua Bengio and etc, it is almost not mentioned (only a reference can be found).

$\endgroup$
  • $\begingroup$ The link to the Bengio paper is broken. Are you referring to Learning long-term dependencies with gradient descent is difficult (Bengio 1994)? $\endgroup$ – user20160 Feb 11 at 2:02
  • $\begingroup$ @user20160 Frankly speaking I am uncertain,, it has been more than two years,,, but I thought you are right. Thanks. $\endgroup$ – Lerner Zhang Feb 11 at 18:22
1
$\begingroup$

I also wonder what does an LSTM have to do with Attractors. Look at the paper "An Empirical Exploration of Recurrent Network Architectures", Proceedings of the 32 nd International Conference on Machine Learning, Lille, France, 2015 by Rafal Jozefowicz, Wojciech Zaremba Ilya Sutskever.

In the 2nd paragraph of Section 2.1 they conclude that LSTMs are not "compatible" with attractor systems based on a result of Bengio:

Bengio et al. (1994) showed that any RNN that stores onebit of information with a stable attractor must necessarilyexhibit a vanishing gradient. As Theorem 4 of Bengio et al.(1994) does not make assumptions on the model architec-ture, it follows that an LSTM would also suffer from van-ishing gradients had it stored information using attractors.But since LSTMs do not suffer from vanishing gradients,they should not be compatible with attractor-based mem-ory systems.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.