I've started reading about Recurrent Neural Networks (RNNs) and Long Short Term Memory (LSTM) ...(...oh, not enough rep points here to list references...)

One thing I don't get: It always seems that neurons in each instance of a hidden layer get "fully connected" with every neuron in the previous instance of the hidden layer, rather than just being connected to the instance of their former self/selves (and maybe a couple others).

Is the fully-connectedness really necessary? Seems like you could save a lot of storage & execution time, and 'lookback' farther in time, if it isn't necessary.

Here's a diagram of my question...

I think this amounts to asking if it's ok to keep only the diagonal (or near-diagonal) elements in the "W^hh" matrix of 'synapses' between the recurring hidden layer. I tried running this using a working RNN code (based on Andrew Trask's demo of binary addition) -- i.e., set all the non-diagonal terms to zero -- and it performed terribly, but keeping terms near the diagonal, i.e. a banded linear system 3 elements wide -- seemed to work as good as the fully-connected version. Even when I increased the sizes of inputs & hidden layer.... So...did I just get lucky?

I found a paper by Lai Wan Chan where he demonstrates that for linear activation functions, it's always possible to reduce a network to "Jordan canonical form" (i.e. the diagonal and nearby elements). But no such proof seems available for sigmoids & other nonlinear activations.

I've also noticed that references to "partially-connected" RNNs just seem mostly to disappear after about 2003, and the treatments I've read from the past few years all seem to assume fully-connectedness. So...why is that?

I've started reading about Recurrent Neural Networks (RNNs) and Long Short Term Memory (LSTM) ...(...oh, not enough rep points here to list references...)

One thing I don't get: It always seems that neurons in each instance of a hidden layer get "fully connected" with every neuron in the previous instance of the hidden layer, rather than just being connected to the instance of their former self/selves (and maybe a couple others).

Is the fully-connectedness really necessary? Seems like you could save a lot of storage & execution time, and 'lookback' farther in time, if it isn't necessary.

Here's a diagram of my question...

I think this amounts to asking if it's ok to keep only the diagonal (or near-diagonal) elements in the "W^hh" matrix of 'synapses' between the recurring hidden layer. I tried running this using a working RNN code (based on Andrew Trask's demo of binary addition) -- i.e., set all the non-diagonal terms to zero -- and it performed terribly, but keeping terms near the diagonal, i.e. a banded linear system 3 elements wide -- seemed to work as good as the fully-connected version. Even when I increased the sizes of inputs & hidden layer.... So...did I just get lucky?

I found a paper by Lai Wan Chan where he demonstrates that for linear activation functions, it's always possible to reduce a network to "Jordan canonical form" (i.e. the diagonal and nearby elements). But no such proof seems available for sigmoids & other nonlinear activations.

I've also noticed that references to "partially-connected" RNNs just seem mostly to disappear after about 2003, and the treatments I've read from the past few years all seem to assume fully-connectedness. So...why is that?

EDIT: 3 years after this question was posted, NVIDIA released this paper, arXiv:1905.12340: "Rethinking Full Connectivity in Recurrent Neural Networks", showing that sparser connections are usually just as accurate and much faster than fully-connected networks. The second diagram above corresponds to the "Diagonal RNN" in the arXiv paper. NVIDIA, I'd be happy to collaborate on future efforts... ;-)

I've started reading about Recurrent Neural Networks (RNNs) and Long Short Term Memory (LSTM) ...(...oh, not enough rep points here to list references...)

One thing I don't get: It always seems that neurons in each instance of a hidden layer get "fully connected" with every neuron in the previous instance of the hidden layer, rather than just being connected to the instance of their former self/selves (and maybe a couple others).

Is the fully-connectedness really necessary? Seems like you could save a lot of storage & execution time, and 'lookback' farther in time, if it isn't necessary.

Here's a diagram of my question...

I think this amounts to asking if it's ok to keep only the diagonal (or near-diagonal) elements in the "W^hh" matrix of 'synapses' between the recurring hidden layer. I tried running this using a working RNN code (based on Andrew Trask's demo of binary addition; not enough rep to post link) -- i.e., set all the non-diagonal terms to zero -- and it performed terribly, but keeping terms near the diagonal, i.e. a banded linear system 3 elements wide -- seemed to work as good as the fully-connected version. Even when I increased the sizes of inputs & hidden layer.... So...did I just get lucky?

I found a paper by Lai Wan Chan where he demonstrates that for linear activation functions, it's always possible to reduce a network to "Jordan canonical form" (i.e. the diagonal and nearby elements). But no such proof seems available for sigmoids & other nonlinear activations.

I've also noticed that references to "partially-connected" RNNs just seem mostly to disappear after about 2003, and the treatments I've read from the past few years all seem to assume fully-connectedness. So...why is that?

I've started reading about Recurrent Neural Networks (RNNs) and Long Short Term Memory (LSTM) ...(...oh, not enough rep points here to list references...)

One thing I don't get: It always seems that neurons in each instance of a hidden layer get "fully connected" with every neuron in the previous instance of the hidden layer, rather than just being connected to the instance of their former self/selves (and maybe a couple others).

Is the fully-connectedness really necessary? Seems like you could save a lot of storage & execution time, and 'lookback' farther in time, if it isn't necessary.

Here's a diagram of my question...

I think this amounts to asking if it's ok to keep only the diagonal (or near-diagonal) elements in the "W^hh" matrix of 'synapses' between the recurring hidden layer. I tried running this using a working RNN code (based on Andrew Trask's demo of binary addition) -- i.e., set all the non-diagonal terms to zero -- and it performed terribly, but keeping terms near the diagonal, i.e. a banded linear system 3 elements wide -- seemed to work as good as the fully-connected version. Even when I increased the sizes of inputs & hidden layer.... So...did I just get lucky?

I found a paper by Lai Wan Chan where he demonstrates that for linear activation functions, it's always possible to reduce a network to "Jordan canonical form" (i.e. the diagonal and nearby elements). But no such proof seems available for sigmoids & other nonlinear activations.

I've also noticed that references to "partially-connected" RNNs just seem mostly to disappear after about 2003, and the treatments I've read from the past few years all seem to assume fully-connectedness. So...why is that?