Formulation of Recurrent Neural Networks

Question

In the book Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville, is said that the classical form of a dynamical system is:

$$s^{t}=f(s^{t-1};\theta)$$

$s^{t}$ is state of system. Therefore, the next state $s^{t}$ is determined through a function $f$ that always uses the same parameters $\theta$ and information about the previous state $s^{t-1}$.

Then it is said:

Consider a dynamical system driven by an external signal $x^{t}$,

$$s^{t}=f(s^{t-1}, x^{t};\theta)$$

where we see that the state now contains information about the whole past sequence.

I don't understand this conceptual passage, what is meant by being driven by an external signal?

$x^{t}$ shouldn't it be the value at time $t$ of the input sequence? Why is it said that the state now contains information about the whole past sequence?

I want to understand in this sense how one comes to define the model of an RNN.

Answer 1

You’re right that $x^t$ is the input sequence’s $t$th element.

Without $\vec{x}$, the dynamical system would continue to evolve over time according to it state and parameters. State $s^t$ would depend only on $s^{t-1}$ (and $\theta$), which in turn would depend on $s^{t-2}$, and so on.

When you start at a given $s^0$, you will always be at the exact same place at time $s^t$.

\begin{align} s^1 &= f(s^0; \theta) \\ s^2 &= f(f(s^0; \theta); \theta) \\ s^3 &= f(f(f(s^0; \theta); \theta); \theta) \\ \vdots \end{align}

---

Instead, $\vec x$ nudges the evolution over time. For a given $s^0$, a different value of $\vec x$ will bring you to a different value of $s^t$. To use your wording, the hidden state will be different depending on the input.

When the book says that it contains information about the whole past sequence, they mean that the state depends on the previous $s^{t-1}$, and so on. It also depends on $\vec x$. You only would get this particular $s^t$ from the particular $\vec x$ that you provided. Changing $\vec x$ changes $\vec s$. \begin{align} s^1 &= f(s^0; x^1; \theta) \\ s^2 &= f(f(s^0, x^1; \theta), x^2; \theta) \\ s^3 &= f(f(f(s^0, x^1; \theta), x^2; \theta), x^3, \theta) \\ \vdots \end{align}

This lends itself to a hand-wavy inductive proof. We know that $s^k$ contains information about $x^k$ and $s^{k-1}$, for any $k$. But $s^{k-1}$ contains information about $x^{k-1}$ for the same reason! So there is information about previous inputs, recursively, included in the current state.