I'm reading this paper about a convolutional neural network (CNN) to model sentences. I think I understand the paper reasonably well until section 3.4. Please consider the following text taken from the paper:
After (dynamic) $k$-max pooling is applied to the result of a convolution, a bias $b\in\mathbb{R}^d$ and a nonlinear function $g$ are applied component-wise to the pooled matrix. There is a single bias value for each row of the pooled matrix. If we temporarily ignore the pooling layer, we may state how one computes each $d$-dimensional column $a$ in the matrix a resulting after the convolutional and non-linear layers. Define $M$ to be the matrix of diagonals: $$M = [diag(m_{:,1}),\ldots,diag(m_{:,m})]$$ where $m$ are the weights of the $d$ filters of the wide convolution. Then after the first pair of a convolutional and a non-linear layer, each column $a$ in the matrix a is obtained as follows, for some index $j$: $$a = g\begin{pmatrix}M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix} + b\end{pmatrix}\tag{6}$$ Here $a$ is the column of first order features. Second order features are similarly obtained by applying Eq. $6$ to a sequence of first order features $a_j, \ldots, a_{j+m' - 1}$ with another weight matrix $M'$. Barring pooling, Eq. $6$ represents a core aspect of the feature extraction function and has a rather general form that we return to below. Together with pooling, the feature function induces position invariance and makes the range of higher-order features variable.
What I know about $M$ is that:
- It is multiplied with a vector that has $m$ rows, so it needs $m$ columns.
- The matrix-vector multiplication results in a column $a$ that is $d$-dimensional, so $M$ must have $d$ rows.
Normally, I would assume that we have: $$M = \begin{bmatrix}m_{1,1} &0&\ldots&0&\ldots&m_{1,m}&0&\ldots&0\\ 0&m_{2,1}&\ldots&0&\ldots&0&m_{2,m}&\ldots&0\\ \vdots&\vdots&\ddots&\vdots&\ldots&\vdots&\vdots&\ddots&\vdots\\ 0&0&\ldots&m_{d,1}&\ldots&0&0&\ldots&m_{d,m}\end{bmatrix}$$
but in this case $M$ has $d\times m$ rows..
Question: What does the matrix $M$ look like?