# Weight matrices computation in attention-based encoder of Deep Learning NLP

On page 4 of this research paper titled A Neural Attention Model for Sentence Summarization , it is mentioned the attention-based encoder is determined by the following formula :

where dimension of the weight matrices are as follow : P is H×(CD), F is H × V, G is D × V. It is mentioned also in line 2 of the above equations that variable p is proportional to exp(x × P × y), which seems to indicate the application of natural exponent and multiplication of the 3 matrices. With that being said, I have 2 main questions.

1.How can the x and P in equation of exp(x × P × y) perform matrix multiplication with each other, since x has the dimension of 1 × (HM) and P has the dimension of H × (CD) ? The column of x doesn't match the row P, so doesn't that mean matrix multiplication isn't possible? Or is the operations of x and P not referring to that of a matrix multiplication?

2.I tried checking the source code here, and I couldn't find the variable that refers to P weight variable that is in the article. The following are the snippet of the attention-based encoder code :

function encoder.build_attnbow_model(opt, data)
print("Encoder model: BoW + Attention")

local D2 = opt.bowDim
local N = opt.window
local V = #data.title_data.dict.index_to_symbol
local V2 = #data.article_data.dict.index_to_symbol

-- Article Embedding.
local article_lookup = nn.LookupTable(V2, D2)()

-- Title Embedding.
local title_lookup = nn.LookupTable(V, D2)()

-- Size Lookup
local size_lookup = nn.Identity()()

-- Ignore size lookup to make NNGraph happy.
local article_context = nn.SelectTable(1)({article_lookup, size_lookup})

-- Pool article
local pad = (opt.attenPool - 1) / 2
local article_match = article_context

-- Title context embedding.
local title_context = nn.View(D2, 1)(
nn.Linear(N * D2, D2)(nn.View(N * D2)(title_lookup)))

-- Attention layer. Distribution over article.
local dot_article_context = nn.MM()({article_match,
title_context})

-- Compute the attention distribution.
local non_linearity = nn.SoftMax()
local attention = non_linearity(nn.Sum(3)(dot_article_context))

local process_article =
nn.Sum(2)(nn.SpatialSubSampling(1, 1, opt.attenPool)(
nn.View(1, -1, D2):setNumInputDims(2)(article_context))))

-- Apply attention to the subsampled article.
local mout = nn.Linear(D2, D2)(
nn.Sum(3)(nn.MM(true, false)(
{process_article,
nn.View(-1, 1):setNumInputDims(1)(attention)})))

-- Apply attention
local encoder_mlp = nn.gModule({article_lookup, size_lookup, title_lookup},
{mout})

encoder_mlp:cuda()
encoder_mlp.lookup = article_lookup.data.module
encoder_mlp.title_lookup = title_lookup.data.module
return encoder_mlp
end


Based on the snippet above, my guess is that the function :

 local dot_article_context = nn.MM()({article_match,
title_context})


compute the result of p by input of x, where x is article match and yc for title_context, and there is no involvement of P weight matrix at all. I feel I misunderstood the concept of attention-based encoder computation, and if that's the case, could somebody clarify this up for me in regards to which part of code that actually does the computation of p?

1.If I understood correctly, x can be formed as an M×H matrix, P is H×(CD), y is (CD)×1, then exp(x×P×y) would give a M×1 matrix, which is used as an unnormalized probability for each word.
2.It seems dot_article_context is the result of x×P×y, and attention (which is obtained by applying softmax to dot_article_context) is the probability p. My guess is that the code uses the same value opt.bowDim for D and H, and the P weight matrix is coded here:
-- Title context embedding.

in the linear layer nn.Linear(N * D2, D2), so title_context should be the result of P×y. In this case computing P×y first instead of x×P is more efficient since x×P will give a larger matrix.