Operation modes in neural turing machine (Graves, 2014)

Question

I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.

More specifically, my questions are about the last step performed by the write head (underlines in green) :

1. What do authors mean by leakage or dispersion ? I feel like adding the sharpening operation should not be required.
2. (Second green part) Why cannot we focus on the exact same location by having a shift of zero ? I am missing something here

Thank you in advance for your help

Answer 1

1. The location address mechanism doesn't focus on a single address, but outputs a distribution on addresses to focus on. This is necessary because otherwise the process would be nondifferentiable, and gradients wouldn't be able to flow backward through the addressing mechanism, and you wouldn't be able to train it. However, if you repeatedly convolve these address distributions, you get something which spreads out more and more over time, which is undesireable behavior.

2. To be concrete, let's say $i$ is the current location, $j$ is proposed by the content addressing, and $k$ is the shift proposed by the location addressing. For simplicity, pretend these are integers, even though both systems actually specify a distribution over addresses. Then the three "modes" (note that all three modes are implemented in a single system shown in Fig 2.) are

   • goto $j$
   • goto $j+k$
   • goto $i+k$

In particular, "focusing on the same location" would be achieved by the third mode, using $k=0$