1
$\begingroup$

On page 4 of this research paper titled A Neural Attention Model for Sentence Summarization , it is mentioned the attention-based encoder is determined by the following formula : enter image description here

where dimension of the weight matrices are as follow : P is H×(CD), F is H × V, G is D × V. It is mentioned also in line 2 of the above equations that variable p is proportional to exp(x × P × y), which seems to indicate the application of natural exponent and multiplication of the 3 matrices. With that being said, I have 2 main questions.

1.How can the x and P in equation of exp(x × P × y) perform matrix multiplication with each other, since x has the dimension of 1 × (HM) and P has the dimension of H × (CD) ? The column of x doesn't match the row P, so doesn't that mean matrix multiplication isn't possible? Or is the operations of x and P not referring to that of a matrix multiplication?

2.I tried checking the source code here, and I couldn't find the variable that refers to P weight variable that is in the article. The following are the snippet of the attention-based encoder code :

function encoder.build_attnbow_model(opt, data)
   print("Encoder model: BoW + Attention")

   local D2 = opt.bowDim
   local N = opt.window
   local V = #data.title_data.dict.index_to_symbol
   local V2 = #data.article_data.dict.index_to_symbol

   -- Article Embedding.
   local article_lookup = nn.LookupTable(V2, D2)()

   -- Title Embedding.
   local title_lookup = nn.LookupTable(V, D2)()

   -- Size Lookup
   local size_lookup = nn.Identity()()

   -- Ignore size lookup to make NNGraph happy.
   local article_context = nn.SelectTable(1)({article_lookup, size_lookup})

   -- Pool article
   local pad = (opt.attenPool - 1) / 2
   local article_match = article_context

   -- Title context embedding.
   local title_context = nn.View(D2, 1)(
      nn.Linear(N * D2, D2)(nn.View(N * D2)(title_lookup)))

   -- Attention layer. Distribution over article.
   local dot_article_context = nn.MM()({article_match,
                                        title_context})

   -- Compute the attention distribution.
   local non_linearity = nn.SoftMax()
   local attention = non_linearity(nn.Sum(3)(dot_article_context))

   local process_article =
      nn.Sum(2)(nn.SpatialSubSampling(1, 1, opt.attenPool)(
                   nn.SpatialZeroPadding(0, 0, pad, pad)(
                      nn.View(1, -1, D2):setNumInputDims(2)(article_context))))

   -- Apply attention to the subsampled article.
   local mout = nn.Linear(D2, D2)(
      nn.Sum(3)(nn.MM(true, false)(
                   {process_article,
                    nn.View(-1, 1):setNumInputDims(1)(attention)})))

   -- Apply attention
   local encoder_mlp = nn.gModule({article_lookup, size_lookup, title_lookup},
      {mout})

   encoder_mlp:cuda()
   encoder_mlp.lookup = article_lookup.data.module
   encoder_mlp.title_lookup = title_lookup.data.module
   return encoder_mlp
end

Based on the snippet above, my guess is that the function :

 local dot_article_context = nn.MM()({article_match,
                                        title_context})

compute the result of p by input of x, where x is article match and yc for title_context, and there is no involvement of P weight matrix at all. I feel I misunderstood the concept of attention-based encoder computation, and if that's the case, could somebody clarify this up for me in regards to which part of code that actually does the computation of p?

Thanks in advance.

$\endgroup$
1
$\begingroup$

1.If I understood correctly, x can be formed as an M×H matrix, P is H×(CD), y is (CD)×1, then exp(x×P×y) would give a M×1 matrix, which is used as an unnormalized probability for each word.

2.It seems dot_article_context is the result of x×P×y, and attention (which is obtained by applying softmax to dot_article_context) is the probability p. My guess is that the code uses the same value opt.bowDim for D and H, and the P weight matrix is coded here:

-- Title context embedding.
local title_context = nn.View(D2, 1)(
   nn.Linear(N * D2, D2)(nn.View(N * D2)(title_lookup)))

in the linear layer nn.Linear(N * D2, D2), so title_context should be the result of P×y. In this case computing P×y first instead of x×P is more efficient since x×P will give a larger matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.