1
$\begingroup$

In this paper (https://arxiv.org/pdf/1508.04025.pdf) local attention is introduced. With local-m attention, we compute the attention vector over the source hidden states within a window around the fixed center $p_t$. This is the same as 'multiplying' the source state sequence with a $rect$-function. And the $rect$-function is not differentiable, however, the authors claim that their model is.

How is this possible?

If this is the case, then why don't we implement hard attention (https://arxiv.org/pdf/1502.03044.pdf) the same way: predict the position $p_t$ with a feed-forward net, instead of sampling, and then just use a window of size 1. This way we can use normal backpropagation and don't need reinforcement learning.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Rect is differentiable in the same way that convolution is differentiable. Even though it assigns 0 weight to all inputs outside of the rectangular window -- in the same way that a convolutional filter assigns 0 weight to all inputs outside of it's receptive field -- the window is slid over the entire range of inputs, so every input contributes to the output.

Hard-attention is differentiable with respect to the inputs to the attention mechanism, but non-differentiable with respect to the parameters inside the attention mechanism. local-m attention does not have this problem, because it does not predict the alignment as a hard-attention mechanism does. Instead local-m attention assumes the input is already perfectly aligned with the output. local-m attention does not predict the position $p_t$. The authors write... "$p_t=t$".

It's worth noting that the other attention model in the paper, local-p attention, is a soft mechanism.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Pardon me, but why do we need the Gaussian functon with local-p attention if $rect$ is differentiable? Is it because local-p is a soft mechanism (= attend to all input locations) and we therefore need to consider positions where the gradient of $rect$ is not defined (at the step from 1 to 0 and vice versa)? $\endgroup$
    – krueger_fl
    Commented Dec 5, 2019 at 7:40
  • $\begingroup$ rect is differentiable with respect to the inputs, but not with respect to the position of the window. $\endgroup$
    – shimao
    Commented Dec 5, 2019 at 8:32
  • $\begingroup$ this is not a problem in local-m because there, the position is not learned (we assume it's perfectly aligned). not so with local-p. $\endgroup$
    – shimao
    Commented Dec 5, 2019 at 8:32
  • $\begingroup$ '$rect$ is differentiable with respect to the inputs, but not with respect to the position of the window.` Can you elaborate on this? Lets say we have $rect(x−p_t)$ with a width of $L$. 1. Does with respect to the inputs mean we derive only in the interval of $L$ where the derivative is 0? 2. How do we even derive with respect to the position $p_t$? I mean I just use $p_t$ in the definition of rect with case differentiation. $\endgroup$
    – krueger_fl
    Commented Dec 6, 2019 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.