What's the difference between feed-forward and recurrent neural networks?

Question

What is the difference between a feed-forward and recurrent neural network?

Why would you use one over the other?

Do other network topologies exist?

Answer 1

Feed-forward ANNs allow signals to travel one way only: from input to output. There are no feedback (loops); i.e., the output of any layer does not affect that same layer. Feed-forward ANNs tend to be straightforward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organisation is also referred to as bottom-up or top-down.

Feedback (or recurrent or interactive) networks can have signals traveling in both directions by introducing loops in the network. Feedback networks are powerful and can get extremely complicated. Computations derived from earlier input are fed back into the network, which gives them a kind of memory. Feedback networks are dynamic; their 'state' is changing continuously until they reach an equilibrium point. They remain at the equilibrium point until the input changes and a new equilibrium needs to be found.

Feedforward neural networks are ideally suitable for modeling relationships between a set of predictor or input variables and one or more response or output variables. In other words, they are appropriate for any functional mapping problem where we want to know how a number of input variables affect the output variable. The multilayer feedforward neural networks, also called multi-layer perceptrons (MLP), are the most widely studied and used neural network model in practice.

As an example of feedback network, I can recall Hopfield’s network. The main use of Hopfield’s network is as associative memory. An associative memory is a device which accepts an input pattern and generates an output as the stored pattern which is most closely associated with the input. The function of the associate memory is to recall the corresponding stored pattern, and then produce a clear version of the pattern at the output. Hopfield networks are typically used for those problems with binary pattern vectors and the input pattern may be a noisy version of one of the stored patterns. In the Hopfield network, the stored patterns are encoded as the weights of the network.

Kohonen’s self-organizing maps (SOM) represent another neural network type that is markedly different from the feedforward multilayer networks. Unlike training in the feedforward MLP, the SOM training or learning is often called unsupervised because there are no known target outputs associated with each input pattern in SOM and during the training process, the SOM processes the input patterns and learns to cluster or segment the data through adjustment of weights (that makes it an important neural network model for dimension reduction and data clustering). A two-dimensional map is typically created in such a way that the orders of the interrelationships among inputs are preserved. The number and composition of clusters can be visually determined based on the output distribution generated by the training process. With only input variables in the training sample, SOM aims to learn or discover the underlying structure of the data.

(The diagrams are from Dana Vrajitoru's C463 / B551 Artificial Intelligence web site.)

Answer 2

What George Dontas writes is correct, however the use of RNNs in practice today is restricted to a simpler class of problems: time series / sequential tasks.

While feedforward networks are used to learn datasets like $(i, t)$ where $i$ and $t$ are vectors, e.g. $i \in \mathcal{R}^n$, for recurrent networks $i$ will always be a sequence, e.g. $i \in (\mathcal{R}^n)^*$.

RNNs have been shown to be able to represent any measureable sequence to sequence mapping by Hammer.

Thus, RNNs are being used nowadays for all kinds of sequential tasks: time series prediction, sequence labeling, sequence classification etc. A good overview can be found on Schmidhuber's page on RNNs.

Answer 3

What is really interesting in asking this question?

Instead of saying RNN and FNN is different in their name. So they are different., I think what is more interesting is in terms of modeling dynamical system, does RNN differ much from FNN?

Background

There has been a debate for modeling dynamical system between Recurrent neural network and Feedforward neural network with additional features as previous time delays (FNN-TD).

From my knowledge after reading those papers on 90's~2010's. The majority of the literature prefer that vanilla RNN is better than FNN in that RNN uses a dynamic memory while FNN-TD is a static memory.

However, there isn't much numerical studies comparing those two. The one [1]on the early showed that for modeling dynamical system, FNN-TD shows comparable performance to vanilla RNN when it is noise-free while perform a bit worse when there is noise. In my experiences on modeling dynamical systems, I often see FNN-TD is good enough.

What is the key difference in how to treat memory effects between RNN and FNN-TD?

Unfortunately, I don't see anywhere and any publication theoretically showed the difference between these two. It is quite interesting. Let's consider a simple case, using a scalar sequence $X_n, X_{n-1},\ldots,X_{n-k}$ to predict $X_{n+1}$. So it is a sequence-to-scalar task.

FNN-TD is the most general, comprehensive way to treating the so called memory effects. Since it is brutal, it covers any kind, any sort, any memory effect theoretically. The only down side is that it just takes too much parameters in practice.

The memory in RNN is nothing but represented as a general "convolution" of the previous information. We all know that convolution between two scalar sequence in general is not an reversible process and deconvolution is most often ill-posed.

My conjecture is "degree of freedom" in such convolution process is determined by the number of hidden units in the RNN state $s$. And it is important for some dynamical systems. Note that the "degree of freedom" can be extended by time delay embedding of states[2] while keeping the same number of hidden units.

Therefore, RNN is actually compressing the previous memory information with loss by doing convolution, while FNN-TD is just exposing them in a sense with no loss of memory information. Note that you can reduce the information loss in convolution by increasing the number of hidden units or using more time delays than vanilla RNN. In this sense, RNN is more flexible than FNN-TD. RNN can achieve no memory loss as FNN-TD and it can be trivial to show the number of parameters are on the same order.

I know someone might want to mention that RNN is carrying the long time effect while FNN-TD can not. For this, I just want to mention that for a continuous autonomous dynamical system, from Takens embedding theory it is a generic property for the embedding to exists for FNN-TD with the seemingly short time memory to achieve the same performance as the seemingly long time memory in RNN. It explains why RNN and FNN-TD does not differ a lot in continuous dynamical system example in the early 90's.

Now I will mention the benefit of RNN. For the task of autonomous dynamical system, using more previous term, although effectively would be the same as using FNN-TD with less previous terms in theory, numerically it would be helpful in that it is more robust to noise. Result in [1] is consistent with this opinion.

Reference

[1]Gençay, Ramazan, and Tung Liu. "Nonlinear modelling and prediction with feedforward and recurrent networks." Physica D: Nonlinear Phenomena 108.1-2 (1997): 119-134.

[2]Pan, Shaowu, and Karthik Duraisamy. "Data-driven Discovery of Closure Models." arXiv preprint arXiv:1803.09318 (2018).