1
$\begingroup$

I read on the "A Compare-Aggregate Model for Matching Text Sequences" model that the "preprocessing layer uses a recurrent neural network":

Our preprocessing layer uses a recurrent neural network to process the two sequences. We use a modified version of LSTM/GRU in which we keep only the input gates for remembering meaningful words: \begin{eqnarray} \nonumber \overline{\mathbf{Q}} & = & \sigma(\mathbf{W}^\text{i} \mathbf{Q} + \mathbf{b}^{\text{i}} \otimes \mathbf{e}_Q) \odot \tanh(\mathbf{W}^{\text{u}}\mathbf{Q}+\mathbf{b}^{\text{u}}\otimes \mathbf{e}_Q), \\ \overline{\mathbf{A}} & = & \sigma(\mathbf{W}^\text{i} \mathbf{A} + \mathbf{b}^{\text{i}} \otimes \mathbf{e}_A) \odot \tanh(\mathbf{W}^{\text{u}}\mathbf{A}+\mathbf{b}^{\text{u}}\otimes \mathbf{e}_A), \end{eqnarray} where $\odot$ is element-wise multiplication, and $\mathbf{W}^\text{i}, \mathbf{W}^\text{u}\in \mathbb{R}^{l\times d}$ and $\mathbf{b}^\text{i},\mathbf{b}^\text{u}\in \mathbb{R}^{l}$ are parameters to be learned. The outer product $(\cdot \otimes \mathbf{e}_X)$ produces a matrix or row vector by repeating the vector or scalar on the left for $X$ times.

The attention layer is built on top of the resulting $\overline{\mathbf{Q}}$ and $\overline{\mathbf{A}}$ as follows: \begin{eqnarray} \nonumber \mathbf{G} & = & \text{softmax} \left( ( \mathbf{W}^{\text{g}} \overline{\mathbf{Q}} + \mathbf{b}^{\text{g}} \otimes \mathbf{e}_Q)^{\text{T}} \overline{\mathbf{A}} \right), \\ \label{eqn:alpha} \mathbf{H} & = & \overline{\mathbf{Q}} \mathbf{G}, \end{eqnarray} where $\mathbf{W}^{\text{g}} \in \mathbb{R}^{l\times l}$ and $\mathbf{b}^{\text{g}} \in \mathbb{R}^{l}$ are parameters to be learned, $\mathbf{G}\in \mathbb{R}^{Q\times A}$ is the attention weight matrix, and $\mathbf{H} \in \mathbb{R}^{l\times A}$ are the attention-weighted vectors. Specifically, $\mathbf{h}_j$, which is the $j^\text{th}$ column vector of $\mathbf{H}$, is a weighted sum of the column vectors of $\overline{\mathbf{Q}}$ and represents the part of $\mathbf{Q}$ that best matches the $j^\text{th}$ word in $\mathbf{A}$. Next we will combine $\mathbf{h}_j$ and $\overline{\mathbf{a}}_j$ using a comparison function. % Here, each column of $\mathbf{H}$ can be treated as the word representation in $\mathbf{Q}$ that is more likely to be aligned with the corresponding column in $\mathbf{H}^\text{a}$. Next, we will make use of these aligned information and build the matched representation between them.

Where is the recurrent neural network?

$\endgroup$
1
$\begingroup$

A comment from the review process has the same concern. The authors posted a revision of the paper here which clarifies the matter: The equations shown for $\overline Q$ and $\overline A$ are not a recurrent network. However, on one of the experimental tasks, they did use an LSTM to compute $\overline Q$ and $\overline A$ instead of using their proposed equation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.