I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.

More specifically, my questions are about the last step performed by the write head (underlines in green) : enter image description here

  1. What do authors mean by leakage or dispersion ? I feel like adding the sharpening operation should not be required.
  2. (Second green part) Why cannot we focus on the exact same location by having a shift of zero ? I am missing something here

Thank you in advance for your help

  1. The location address mechanism doesn't focus on a single address, but outputs a distribution on addresses to focus on. This is necessary because otherwise the process would be nondifferentiable, and gradients wouldn't be able to flow backward through the addressing mechanism, and you wouldn't be able to train it. However, if you repeatedly convolve these address distributions, you get something which spreads out more and more over time, which is undesireable behavior.

  2. To be concrete, let's say $i$ is the current location, $j$ is proposed by the content addressing, and $k$ is the shift proposed by the location addressing. For simplicity, pretend these are integers, even though both systems actually specify a distribution over addresses. Then the three "modes" (note that all three modes are implemented in a single system shown in Fig 2.) are

    • goto $j$
    • goto $j+k$
    • goto $i+k$

In particular, "focusing on the same location" would be achieved by the third mode, using $k=0$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.