Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am new to Recurrent Neural Networks(RNN) and still learning the concepts. I understand at an abstract level that an Echo State Network(ESN) is able to (re)produce a sequence of inputs, i.e. a signal, even after the input has been removed. However, I found the Scholarpedia article too difficult to grasp and understand completely.
Can someone please explain how the learning works mathematically in the simplest possible form.
An Echo State Network is an instance of the more general concept of Reservoir Computing. The basic idea behind the ESN is to get the benefits of a RNN (process a sequence of inputs that are dependent on each other, i.e. time dependencies like a signal) but without the problems of training a traditional RNN like the vanishing gradient problem.
ESNs achieve this by having a relatively large reservoir of sparsely connected neurons using a sigmoidal transfer function (relative to input size, something like 100-1000 units). The connections in the reservoir are assigned once and are completely random; the reservoir weights do not get trained. Input neurons are connected to the reservoir and feed the input activations into the reservoir - these too are assigned untrained random weights. The only weights that are trained are the output weights which connect the reservoir to the output neurons.
In training, the inputs will be fed to the reservoir and a teacher output will be applied to the output units. The reservoir states are captured over time and stored. Once all of the training inputs have been applied, a simple application of linear regression can be used between the captured reservoir states and the target outputs. These output weights can then be incorporated into the existing network and used for novel inputs.
The idea is that the sparse random connections in the reservoir allow previous states to "echo" even after they have passed, so that if the network receives a novel input that is similar to something it trained on, the dynamics in the reservoir will start to follow the activation trajectory appropriate for the input and in that way can provide a matching signal to what it trained on, and if it is well-trained it will be able to generalize from what it has already seen, following activation trajectories that would make sense given the input signal driving the reservoir.
The advantage of this approach is in the incredibly simple training procedure since most of the weights are assigned only once and at random. Yet they are able to capture complex dynamics over time and are able to model properties of dynamical systems. By far the most helpful papers I have found on ESNs are:
A tutorial on training RNNs by Herbert Jaeger (curator of the Scholarpedia page on ESNs)
A Practical Guide to Applying Echo State Networks by Mantas Lukoševičius
They both have easy to understand explanations to go along with the formalism and outstanding advice for creating an implementation with guidance for choosing appropriate parameter values.
UPDATE: The Deep Learning book from Goodfellow, Bengio, and Courville has a slightly more detailed but still nice high-level discussion of Echo State Networks. Section 10.7 discusses the vanishing (and exploding) gradient problem and the difficulties of learning long-term dependencies. Section 10.8 is all about Echo State Networks. It specifically goes into detail about why it's crucial to select reservoir weights that have an appropriate spectral radius value - it works together with the nonlinear activation units to encourage stability while still propagating information through time.
Learning in an ESN isn’t primary forced to adapting weights, more respectively the output layer learns which output to produce for the current state the network has. The internal state is based on network dynamics and is called dynamic reservoir state. To understand how the reservoir states shape out, we need to look at the topology of an ESN.
The input unit(s) are connected to neurons in the internal units (reservoir units), the weights are randomly initialized. The reservoir units are randomly and sparsely connected and as well have random weights. The output unit is also connected to all reservoir units thus receives the reservoir state and produces a corresponding output.
The input activation raises the network dynamics. The signal floats $t$ timesteps through the recurrent connected reservoir units. You can imagine it as an echo reoccurring $t$ times in the net (which gets distorted). The only weights which get adapted are the weights to the output unit. This means, that the output layer learns which output has to belong to a given reservoir state. That also means the training becomes a linear regression task.
Before we can explain how the training works in detail we have to explain and define some things:
Teacher Forcing means feed time series input into network as well as corresponding desired output (time delayed). To feed the desired output of $T$ at $t$ back is called output feedback. We therefore need some randomly initialized weights stored in the matrix $W_{fb}$. In figure 1 those edges are displayed with dotted arrows.
Variable definitons:
Finally how does the training work in detail?
Because learning is very fast we can try out many network topologies to get one which fits well.
To measure the performance of an ESN:
Spectral Radius and ESN
Some smart people have proven, that the Echo State Property of an ESN may only be given if the Spec-tral Radius of the reservoir weight matrix is smaller or equal than $1$. The Echo State Property means the system forgets its inputs after a limited amount of time. This property is necessary for an ESN to not ex-plode in activity and to be able to learn.
