I've made some progress towards coding Neural Turing Machines (NTMs), one of the most interesting new twists on neural nets. As usual, however, I'm getting tripped up on something very basic instead of its more novel aspects, like the use of fuzzy read-write heads to retrieve information from location and content-based addresses. The obstacle I've run into probably has an easy answer: how are the read-write heads and memory addresses actually attached to the controller modules?
In the original paper and several follow-up studies, ordinary feed-forward nets and Long-Term Short-Term (LSTM) modules serve as the controllers. As the diagram below (from p. 5 of the original paper written by Google DeepMind specialists in late 20141) illustrates, the topology is fairly simple if you view it from the highest level of abstraction: inputs go into the feed-forward or LSTM modules that serve as the controller, which interface back and forth with the read-write heads, then produce outputs of some kind. All of this is quite clear - except for how to actually attach the controllers to the read-write heads.
For example, in an ordinary neural net, one layer can interact with another through direct connections between neurons, including forward, backwards, lateral and recurrent connection types, as well as more advanced variants like group and higher-order connections. Various formulas can then be used to update the weights, activations, etc. based in part on the values found in layers connected to them. The neurons of one layer may also interact with another layer by reading global values, such as cost function results or some other aggregate for an entire layer, then adjust their activations and/or weights accordingly. How the individual neurons and connections in the feed-forward, LSTM and other controller types actually incorporate information from the read-write heads is glossed over in all of the research I've read on the topic to date. A surprising amount of it simply regurgitates the same high-level diagram above, such as p. 19 of this slideshow and p. 5 of this similar presentation. More advanced topologies can be found in this paper on Structured Memory for Neural Turing Machines, using multiple read/write heads, memories and LSTM Modules, but they begin with the same basic module-level depiction, without any explanation of how to wire these modules together. These papers on Evolving Neural Turing Machines and One-Shot Learning with Memory-Augmented Neural Networks were of little help for this particular issue. This study uses a reinforcement learning scheme, but there's no way of telling whether its NTMs use a completely different interface to their controllers, especially since the earlier research isn't clear. Likewise, this research published on Dynamic Neural Turing Machines with Soft and Hard Addressing Schemes seems to refer to a more advanced variant structure, which means it is likewise of no aid in interpreting the original, bare-bones versions of NTMs. In short, all of the articles I've read to date have been fascinating and well-written, but none have directly addressed this aspect of design.
Can anyone explain how the read-write heads exchange information with the neurons and connections of the controller? It is not the memory structures I'm confused about, so there's no need to rehash the algorithms from pp. 7-9 in the Graves et al.'s original paper, unless I'm missing something in them. I also grasp how ordinary feed-forward nets and LSTMs work, so I don't need a recap of those topics either. I'm just wondering what possible formulas or low-level topological structures can be used to transfer the values calculated in the memory structures into the controller and affect the activations and weights (or vice-versa), which in turn will determine the final output. This seems to be treated as a given in the literature, but it's ironically the part that went furthest over my head.
1Graves, Alex; Wayne, Greg and Danihelka, Ivo, 2014, Neural Turing Machines. Cornell Univeristy Library arXiv.org Repository: Ithaca, New York