I remember recently seeing or reading a paper about a new type of recurrent neural network that enabled long term memory over sequences by having only part of the neurons active at any given timestep.

The timestep unrolled connections looked something like this (possibly not exactly like this)

enter image description here

This was done to model time series, and enabled the network to remember events that happened both a very long time ago, and a very short time ago.

Does anyone know the name of this type of network architecture?

  • $\begingroup$ Do you remember which paper it was? $\endgroup$ – Franck Dernoncourt Mar 3 '17 at 22:47
  • $\begingroup$ @FranckDernoncourt No, that's why I'm asking! I do remember it tried to model events both at a long time ago, and a very short time ago $\endgroup$ – Quizzler net Mar 3 '17 at 23:12
  • $\begingroup$ the image reminds me of arxiv.org/pdf/1603.03827.pdf fig 2 but it's not an RNN there (though we could've used one) $\endgroup$ – Franck Dernoncourt Mar 3 '17 at 23:16

This type of architecture is referred to as a "Dilated Causal Convolution".

As you mentioned, the effect in timeseries data is that the receptive field of the higher layers extend well beyond their immediate receptive field.

Check out Google's paper on Wavenet for more details.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.