1
$\begingroup$

I'm reading this paper about a convolutional neural network (CNN) to model sentences. I think I understand the paper reasonably well until section 3.4. Please consider the following text taken from the paper:

After (dynamic) $k$-max pooling is applied to the result of a convolution, a bias $b\in\mathbb{R}^d$ and a nonlinear function $g$ are applied component-wise to the pooled matrix. There is a single bias value for each row of the pooled matrix. If we temporarily ignore the pooling layer, we may state how one computes each $d$-dimensional column $a$ in the matrix a resulting after the convolutional and non-linear layers. Define $M$ to be the matrix of diagonals: $$M = [diag(m_{:,1}),\ldots,diag(m_{:,m})]$$ where $m$ are the weights of the $d$ filters of the wide convolution. Then after the first pair of a convolutional and a non-linear layer, each column $a$ in the matrix a is obtained as follows, for some index $j$: $$a = g\begin{pmatrix}M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix} + b\end{pmatrix}\tag{6}$$ Here $a$ is the column of first order features. Second order features are similarly obtained by applying Eq. $6$ to a sequence of first order features $a_j, \ldots, a_{j+m' - 1}$ with another weight matrix $M'$. Barring pooling, Eq. $6$ represents a core aspect of the feature extraction function and has a rather general form that we return to below. Together with pooling, the feature function induces position invariance and makes the range of higher-order features variable.

What I know about $M$ is that:

  • It is multiplied with a vector that has $m$ rows, so it needs $m$ columns.
  • The matrix-vector multiplication results in a column $a$ that is $d$-dimensional, so $M$ must have $d$ rows.

Normally, I would assume that we have: $$M = \begin{bmatrix}m_{1,1} &0&\ldots&0&\ldots&m_{1,m}&0&\ldots&0\\ 0&m_{2,1}&\ldots&0&\ldots&0&m_{2,m}&\ldots&0\\ \vdots&\vdots&\ddots&\vdots&\ldots&\vdots&\vdots&\ddots&\vdots\\ 0&0&\ldots&m_{d,1}&\ldots&0&0&\ldots&m_{d,m}\end{bmatrix}$$

but in this case $M$ has $d\times m$ rows..

Question: What does the matrix $M$ look like?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Per the top of p. 658 in the above linked article, $w_i\in\mathbb{R}^d$. So $\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$ is $d*m$ by 1, and your "normal assumption" about $M$ is correct, i.e., that M is $d$ by $d*m$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your answer Mark! But if $\begin{bmatrix}w_j\ldots w_{j+m - 1}\end{bmatrix}$ is $d\times m$ then $a$ is not a vector but a matrix right? Furthermore you say that $M$ is $d$ by $d\times m$; does that mean that $M$ is a tensor? I'm confused because in the paper $M$ is called a matrix right? $\endgroup$
    – Mr. President
    May 13, 2018 at 17:40
  • $\begingroup$ Each $w_i$ is a $d$ by $1$ vector. Then $w_j$ ... $w_{j+m-1}$ are stacked vertically into a $d*m$ by $1$ vector $\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$. As for $M$, it is a $d$ by $d*m$ matrix, which is laid out as you depicted. All the dimensions are consistent. $\endgroup$
    – Mark L. Stone
    May 13, 2018 at 17:56
  • $\begingroup$ But if you multiply $M\begin{bmatrix}w_j\\\ldots\\w_{j+m-1}\end{bmatrix}$ then you get a matrix right? Which would mean that $a$ is a matrix as well. However, the paper states that it is a vector. I'm obviously missing something sorry! $\endgroup$
    – Mr. President
    May 13, 2018 at 18:05
  • $\begingroup$ $M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$ is the multiplication of a $d$ by $d*m$ matrix by a $d*m$ by $1$ vector, which results in a $d$ by $1$ vector. $\endgroup$
    – Mark L. Stone
    May 13, 2018 at 18:10
  • $\begingroup$ Could you bit a bit more elaborate? Why is $M$ a $d$ by $d\times m$ matrix? The weights $m_{:,j}$ of the filters $d$ are just vectors right? So why is $M$ a 3 dimensional matrix if it is just composed of vectors? $\endgroup$
    – Mr. President
    May 13, 2018 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.