Purpose of a 'time constant' in a recurrent neural network?

Question

I have been trying to understand both the reasoning and method of programming/implementing a network as described in this paper:

Running Across the Reality Gap: Octopod Locomotion Evolved in a Minimal Simulation

Specifically in equation (1) they specify the use of a "time constant" $T_j$:

(1) $T_jA'_j = I_j-A_j + \sum_iW_{ij}O_i$

(2) $O_j = \dfrac{1}{1 - e^{\mathrm{threshold} - A_j}}$

I don't understand why $T_jA'_j$ is on the left hand side of (1)? Also in (2) should that not be $A'_j$ instead of $A_j$.

I don't understand how $T_j$ the time constant is supposed to be used in the network or it's purpose as opposed to using a typical weighted network. So if I was to attempt to program it how do you use it in evaluating the network in sequence?

Given it's a recurrent neural network it could be related to the timing of evaluation of neurons in the network (i.e. the feedback of the hidden neurons to themselves) but it isn't clear at all.

Also I assume the $-A_j$ term in (1) is effectively a connection of the neuron back to itself with a $-1$ weight? Is there some reasoning why feedback of the negative of the previous state is a good idea?

And that (2) is some variant of a sigmoid activation with a bias used for some reason.

From looking at the Wikipedia definition of Time constant, it seems that it could be some sort of circuit delay/time-based-falloff but I am not sure how that would be realistically implemented in software?

Answer 1

It looks like this is a system of ordinary differential equations (ODEs) in time $t$, where equation (1) corresponds to $$T_j\frac{dA_j}{dt} = I_j-A_j[t] + \sum_iW_{ij}O_i[t]$$ and equation (2) corresponds to $$O_j[t] = \sigma(\,A_j[t]-\mathrm{threshold}\,)$$ where $$\sigma(x)=\dfrac{1}{1 - e^{-x}}$$ is the standard logistic curve, also known as "sigmoid activation", in the (artificial?) neural network community.

This is a nonlinear system of first order ODEs, but the essence of the "time constant" can be understood from the following simplified form $$\tau\frac{dx}{dt}=c-x[t]$$ Here the system will asymptotically approach the steady state $x=c$, such that for initial condition $x[0]=x_0$ we have $$x[t]=c+(x_0-c)e^{-t/\tau}$$ From these equations, we can see that the time constant $\tau$ gives the timescale of evolution \begin{align} t\ll\tau &\implies x[t]\approx x_0 \\ t\gg\tau &\implies x[t]\approx c \end{align} In this simple example with scalar $x$ and $\tau$, the time constant is not particularly important, as it essentially just changes the "units" of time measurement (e.g. seconds vs. years).

In your ODE system, with vector $\vec{A}$ and $\vec{T}$, the time constants can be much more significant, as they can represent variations in the evolution timescales of different components of $\vec{A}$. For example, if $T_1=10^{-3}$ and $T_2=10^3$, then $A_1$ will be very "fast" relative to $A_2$.

(An ODE system with widely varying evolution timescales is called stiff. A nice discussion of this can be found on the SciComp SE site here.)

As for the second question, the $I_j-A_j$ term is exactly analogous to my simplified example above. That is, the linear part of each ODE in (1) represents an asymptotic approach of $A_j[t]$ to the steady state $I_j$. Notice that this term is uncoupled as well as linear. So in the absence of feedback from other "neurons" $A_{j\neq i}$, each neuron will go to a steady-state background $A_j=I_j$.