3
$\begingroup$

I've made some progress towards coding Neural Turing Machines (NTMs), one of the most interesting new twists on neural nets. As usual, however, I'm getting tripped up on something very basic instead of its more novel aspects, like the use of fuzzy read-write heads to retrieve information from location and content-based addresses. The obstacle I've run into probably has an easy answer: how are the read-write heads and memory addresses actually attached to the controller modules?

In the original paper and several follow-up studies, ordinary feed-forward nets and Long-Term Short-Term (LSTM) modules serve as the controllers. As the diagram below (from p. 5 of the original paper written by Google DeepMind specialists in late 20141) illustrates, the topology is fairly simple if you view it from the highest level of abstraction: inputs go into the feed-forward or LSTM modules that serve as the controller, which interface back and forth with the read-write heads, then produce outputs of some kind. All of this is quite clear - except for how to actually attach the controllers to the read-write heads.

A basic high-level NTM Topology

For example, in an ordinary neural net, one layer can interact with another through direct connections between neurons, including forward, backwards, lateral and recurrent connection types, as well as more advanced variants like group and higher-order connections. Various formulas can then be used to update the weights, activations, etc. based in part on the values found in layers connected to them. The neurons of one layer may also interact with another layer by reading global values, such as cost function results or some other aggregate for an entire layer, then adjust their activations and/or weights accordingly. How the individual neurons and connections in the feed-forward, LSTM and other controller types actually incorporate information from the read-write heads is glossed over in all of the research I've read on the topic to date. A surprising amount of it simply regurgitates the same high-level diagram above, such as p. 19 of this slideshow and p. 5 of this similar presentation. More advanced topologies can be found in this paper on Structured Memory for Neural Turing Machines, using multiple read/write heads, memories and LSTM Modules, but they begin with the same basic module-level depiction, without any explanation of how to wire these modules together. These papers on Evolving Neural Turing Machines and One-Shot Learning with Memory-Augmented Neural Networks were of little help for this particular issue. This study uses a reinforcement learning scheme, but there's no way of telling whether its NTMs use a completely different interface to their controllers, especially since the earlier research isn't clear. Likewise, this research published on Dynamic Neural Turing Machines with Soft and Hard Addressing Schemes seems to refer to a more advanced variant structure, which means it is likewise of no aid in interpreting the original, bare-bones versions of NTMs. In short, all of the articles I've read to date have been fascinating and well-written, but none have directly addressed this aspect of design.

Can anyone explain how the read-write heads exchange information with the neurons and connections of the controller? It is not the memory structures I'm confused about, so there's no need to rehash the algorithms from pp. 7-9 in the Graves et al.'s original paper, unless I'm missing something in them. I also grasp how ordinary feed-forward nets and LSTMs work, so I don't need a recap of those topics either. I'm just wondering what possible formulas or low-level topological structures can be used to transfer the values calculated in the memory structures into the controller and affect the activations and weights (or vice-versa), which in turn will determine the final output. This seems to be treated as a given in the literature, but it's ironically the part that went furthest over my head.

1Graves, Alex; Wayne, Greg and Danihelka, Ivo, 2014, Neural Turing Machines. Cornell Univeristy Library arXiv.org Repository: Ithaca, New York

$\endgroup$
  • $\begingroup$ The diagram appears to suggest there are cycles in the net, that would imply that this is a recurrent neural net. Why couldn't there just be regular connection from the controller to the read-write head and back again? $\endgroup$ – dimpol Oct 20 '16 at 8:04
  • $\begingroup$ @dimpol Because a "regular connection" would have to go to a regular neuron, but the read-write heads don't contain any. They compute what amounts to a fuzzy weighting over matrix. How those values are incorporated into the activations and weights of a standard neural net isn't really clear. $\endgroup$ – SQLServerSteve Oct 20 '16 at 15:23
2
$\begingroup$

I can precisely relate to your frustration, since for the past 2-3 days, I've been scratching my head with the exact same problem. Every blog post and follow-up paper that I read, just copy-pasted the exact same block diagram, with more focus on the addressing scheme, which to be honest is pretty straightforward.

Technically, it was clear to me that you can just use the hidden state or the memory of the LSTM, and pass it through some network to produce the add, erase vectors, and the weights for the soft attention, but no paper ever confirmed it.

However, if you'll look at the appendix of the " One-Shot Learning with Memory-Augmented Neural Networks" paper, you'll notice that they briefly mention that they use the LSTM's memory state as the key, as well as the value to write in the memory. So, there you go, this paper sort of confirms that you use the memory state of the LSTM to connect the controller with the read and write heads.

$\endgroup$
  • 1
    $\begingroup$ I'll check into this, thanks for the heads-up. On questions like this with few reads and little-known subject matter I tend to wait awhile before marking an answer, just to make sure that other useful answers don't pop up weeks later. $\endgroup$ – SQLServerSteve Jun 28 '17 at 9:43
  • 1
    $\begingroup$ Sure, fair enough! To be honest, it's very easy to miss this detail in the aforementioned paper. Just search for "For reading, the controller cell state serves as a query for $M_t$." in the supplementary material/appendix of the paper. $\endgroup$ – Shatu Jun 29 '17 at 6:30
  • 1
    $\begingroup$ Great answer. Just for the sake of completeness, I'll add that in the design particular to this paper, they perform some concatenation operations with the internal memory state when reading and writing. I found it helpful to read a little about that as well (which may be specific just to this One-Shot algorithm, but is probably indicative of the manner in which other NTMs read/write to their controllers). $\endgroup$ – SQLServerSteve Sep 6 '17 at 20:11
0
$\begingroup$

Perhaps you should try this type of locality sensitive hash (LSH) based associative memory. Associative Memory It seems like a simpler and better solution.

$\endgroup$
  • $\begingroup$ @SSRegen - Thanks for the suggestion, but this does not address the question, which was narrowly focused on determining the internal structure of a very specific type of neural net. You're a new poster, so just a word to the wise: Most StackExchange threads are quite narrow & general suggestions outside the question's scope will often be downvoted. It's considered acceptable to put them in comments (as a new user, you may not be able to post comments yet). Links also require more thorough explanation in case the pages disappear. Welcome to CV & thanks for the well-intentioned feedback. $\endgroup$ – SQLServerSteve Apr 5 '19 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.