1
$\begingroup$

I'm reading this paper about a convolutional neural network (CNN) to model sentences. I think I understand the paper reasonably well until section 3.4. Please consider the following text taken from the paper:

After (dynamic) $k$-max pooling is applied to the result of a convolution, a bias $b\in\mathbb{R}^d$ and a nonlinear function $g$ are applied component-wise to the pooled matrix. There is a single bias value for each row of the pooled matrix. If we temporarily ignore the pooling layer, we may state how one computes each $d$-dimensional column $a$ in the matrix a resulting after the convolutional and non-linear layers. Define $M$ to be the matrix of diagonals: $$M = [diag(m_{:,1}),\ldots,diag(m_{:,m})]$$ where $m$ are the weights of the $d$ filters of the wide convolution. Then after the first pair of a convolutional and a non-linear layer, each column $a$ in the matrix a is obtained as follows, for some index $j$: $$a = g\begin{pmatrix}M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix} + b\end{pmatrix}\tag{6}$$ Here $a$ is the column of first order features. Second order features are similarly obtained by applying Eq. $6$ to a sequence of first order features $a_j, \ldots, a_{j+m' - 1}$ with another weight matrix $M'$. Barring pooling, Eq. $6$ represents a core aspect of the feature extraction function and has a rather general form that we return to below. Together with pooling, the feature function induces position invariance and makes the range of higher-order features variable.

What I know about $M$ is that:

  • It is multiplied with a vector that has $m$ rows, so it needs $m$ columns.
  • The matrix-vector multiplication results in a column $a$ that is $d$-dimensional, so $M$ must have $d$ rows.

Normally, I would assume that we have: $$M = \begin{bmatrix}m_{1,1} &0&\ldots&0&\ldots&m_{1,m}&0&\ldots&0\\ 0&m_{2,1}&\ldots&0&\ldots&0&m_{2,m}&\ldots&0\\ \vdots&\vdots&\ddots&\vdots&\ldots&\vdots&\vdots&\ddots&\vdots\\ 0&0&\ldots&m_{d,1}&\ldots&0&0&\ldots&m_{d,m}\end{bmatrix}$$

but in this case $M$ has $d\times m$ rows..

Question: What does the matrix $M$ look like?

$\endgroup$
1
$\begingroup$

Per the top of p. 658 in the above linked article, $w_i\in\mathbb{R}^d$. So $\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$ is $d*m$ by 1, and your "normal assumption" about $M$ is correct, i.e., that M is $d$ by $d*m$.

$\endgroup$
  • $\begingroup$ Thanks for your answer Mark! But if $\begin{bmatrix}w_j\ldots w_{j+m - 1}\end{bmatrix}$ is $d\times m$ then $a$ is not a vector but a matrix right? Furthermore you say that $M$ is $d$ by $d\times m$; does that mean that $M$ is a tensor? I'm confused because in the paper $M$ is called a matrix right? $\endgroup$ – Mr. President May 13 '18 at 17:40
  • $\begingroup$ Each $w_i$ is a $d$ by $1$ vector. Then $w_j$ ... $w_{j+m-1}$ are stacked vertically into a $d*m$ by $1$ vector $\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$. As for $M$, it is a $d$ by $d*m$ matrix, which is laid out as you depicted. All the dimensions are consistent. $\endgroup$ – Mark L. Stone May 13 '18 at 17:56
  • $\begingroup$ But if you multiply $M\begin{bmatrix}w_j\\\ldots\\w_{j+m-1}\end{bmatrix}$ then you get a matrix right? Which would mean that $a$ is a matrix as well. However, the paper states that it is a vector. I'm obviously missing something sorry! $\endgroup$ – Mr. President May 13 '18 at 18:05
  • $\begingroup$ $M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$ is the multiplication of a $d$ by $d*m$ matrix by a $d*m$ by $1$ vector, which results in a $d$ by $1$ vector. $\endgroup$ – Mark L. Stone May 13 '18 at 18:10
  • $\begingroup$ Could you bit a bit more elaborate? Why is $M$ a $d$ by $d\times m$ matrix? The weights $m_{:,j}$ of the filters $d$ are just vectors right? So why is $M$ a 3 dimensional matrix if it is just composed of vectors? $\endgroup$ – Mr. President May 13 '18 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.