How are Controllers Attached to Read-Write Heads in Neural Turing Machines? I've made some progress towards coding Neural Turing Machines (NTMs), one of the most interesting new twists on neural nets. As usual, however, I'm getting tripped up on something very basic instead of its more novel aspects, like the use of fuzzy read-write heads to retrieve information from location and content-based addresses.  The obstacle I've run into probably has an easy answer: how are the read-write heads and memory addresses actually attached to the controller modules? 
In the original paper and several follow-up studies, ordinary feed-forward nets and Long-Term Short-Term (LSTM) modules serve as the controllers. As the diagram below (from p. 5 of the original paper written by Google DeepMind specialists in late 20141) illustrates, the topology is fairly simple if you view it from the highest level of abstraction: inputs go into the feed-forward or LSTM modules that serve as the controller, which interface back and forth with the read-write heads, then produce outputs of some kind. All of this is quite clear - except for how to actually attach the controllers to the read-write heads. 

For example, in an ordinary neural net, one layer can interact with another through direct connections between neurons, including forward, backwards, lateral and recurrent connection types, as well as more advanced variants like group and higher-order connections. Various formulas can then be used to update the weights, activations, etc. based in part on the values found in layers connected to them. The neurons of one layer may also interact with another layer by reading global values, such as cost function results or some other aggregate for an entire layer, then adjust their activations and/or weights accordingly. How the individual neurons and connections in the feed-forward, LSTM and other controller types actually incorporate information from the read-write heads is glossed over in all of the research I've read on the topic to date. A surprising amount of it simply regurgitates the same high-level diagram above, such as p. 19 of this slideshow  and p. 5 of this similar presentation. More advanced topologies can be found in this paper on Structured Memory for Neural Turing Machines, using multiple read/write heads, memories and LSTM Modules, but they begin with the same basic module-level depiction, without any explanation of how to wire these modules together. These papers on Evolving Neural Turing Machines and One-Shot Learning with Memory-Augmented Neural Networks were of little help for this particular issue. This study uses a reinforcement learning scheme, but there's no way of telling whether its NTMs use a completely different interface to their controllers, especially since the earlier research isn't clear. Likewise, this research published on Dynamic Neural Turing Machines with Soft and Hard Addressing Schemes seems to refer to a more advanced variant structure, which means it is likewise of no aid in interpreting the original, bare-bones versions of NTMs. In short, all of the articles I've read to date have been fascinating and well-written, but none have directly addressed this aspect of design.
Can anyone explain how the read-write heads exchange information with the neurons and connections of the controller? It is not the memory structures I'm confused about, so there's no need to rehash the algorithms from pp. 7-9 in the Graves et al.'s original paper, unless I'm missing something in them. I also grasp how ordinary feed-forward nets and LSTMs work, so I don't need a recap of those topics either. I'm just wondering what possible formulas or low-level topological structures can be used to transfer the values calculated in the memory structures into the controller and affect the activations and weights (or vice-versa), which in turn will determine the final output. This seems to be treated as a given in the literature, but it's ironically the part that went furthest over my head. 
1Graves, Alex; Wayne, Greg and Danihelka, Ivo, 2014, Neural Turing Machines. Cornell Univeristy Library arXiv.org Repository: Ithaca, New York    
 A: I can precisely relate to your frustration, since for the past 2-3 days, I've been scratching my head with the exact same problem. Every blog post and follow-up paper that I read, just copy-pasted the exact same block diagram, with more focus on the addressing scheme, which to be honest is pretty straightforward.
Technically, it was clear to me that you can just use the hidden state or the memory of the LSTM, and pass it through some network to produce the add, erase vectors, and the weights for the soft attention, but no paper ever confirmed it.
However, if you'll look at the appendix of the " One-Shot Learning with Memory-Augmented Neural Networks" paper, you'll notice that they briefly mention that they use the LSTM's memory state as the key, as well as the value to write in the memory. So, there you go, this paper sort of confirms that you use the memory state of the LSTM to connect the controller with the read and write heads.
A: Perhaps you should try this type of locality sensitive hash (LSH) based associative memory.
Associative Memory
It seems like a simpler and better solution.
