# What does the matrix $M = [diag(m_{:,1}),\ldots,diag(m_{:,m})]$ look like?

I'm reading this paper about a convolutional neural network (CNN) to model sentences. I think I understand the paper reasonably well until section 3.4. Please consider the following text taken from the paper:

After (dynamic) $k$-max pooling is applied to the result of a convolution, a bias $b\in\mathbb{R}^d$ and a nonlinear function $g$ are applied component-wise to the pooled matrix. There is a single bias value for each row of the pooled matrix. If we temporarily ignore the pooling layer, we may state how one computes each $d$-dimensional column $a$ in the matrix a resulting after the convolutional and non-linear layers. Define $M$ to be the matrix of diagonals: $$M = [diag(m_{:,1}),\ldots,diag(m_{:,m})]$$ where $m$ are the weights of the $d$ filters of the wide convolution. Then after the first pair of a convolutional and a non-linear layer, each column $a$ in the matrix a is obtained as follows, for some index $j$: $$a = g\begin{pmatrix}M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix} + b\end{pmatrix}\tag{6}$$ Here $a$ is the column of first order features. Second order features are similarly obtained by applying Eq. $6$ to a sequence of first order features $a_j, \ldots, a_{j+m' - 1}$ with another weight matrix $M'$. Barring pooling, Eq. $6$ represents a core aspect of the feature extraction function and has a rather general form that we return to below. Together with pooling, the feature function induces position invariance and makes the range of higher-order features variable.

What I know about $M$ is that:

• It is multiplied with a vector that has $m$ rows, so it needs $m$ columns.
• The matrix-vector multiplication results in a column $a$ that is $d$-dimensional, so $M$ must have $d$ rows.

Normally, I would assume that we have: $$M = \begin{bmatrix}m_{1,1} &0&\ldots&0&\ldots&m_{1,m}&0&\ldots&0\\ 0&m_{2,1}&\ldots&0&\ldots&0&m_{2,m}&\ldots&0\\ \vdots&\vdots&\ddots&\vdots&\ldots&\vdots&\vdots&\ddots&\vdots\\ 0&0&\ldots&m_{d,1}&\ldots&0&0&\ldots&m_{d,m}\end{bmatrix}$$

but in this case $M$ has $d\times m$ rows..

Question: What does the matrix $M$ look like?

## 1 Answer

Per the top of p. 658 in the above linked article, $w_i\in\mathbb{R}^d$. So $\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$ is $d*m$ by 1, and your "normal assumption" about $M$ is correct, i.e., that M is $d$ by $d*m$.

• Thanks for your answer Mark! But if $\begin{bmatrix}w_j\ldots w_{j+m - 1}\end{bmatrix}$ is $d\times m$ then $a$ is not a vector but a matrix right? Furthermore you say that $M$ is $d$ by $d\times m$; does that mean that $M$ is a tensor? I'm confused because in the paper $M$ is called a matrix right? – Mr. President May 13 '18 at 17:40
• Each $w_i$ is a $d$ by $1$ vector. Then $w_j$ ... $w_{j+m-1}$ are stacked vertically into a $d*m$ by $1$ vector $\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$. As for $M$, it is a $d$ by $d*m$ matrix, which is laid out as you depicted. All the dimensions are consistent. – Mark L. Stone May 13 '18 at 17:56
• But if you multiply $M\begin{bmatrix}w_j\\\ldots\\w_{j+m-1}\end{bmatrix}$ then you get a matrix right? Which would mean that $a$ is a matrix as well. However, the paper states that it is a vector. I'm obviously missing something sorry! – Mr. President May 13 '18 at 18:05
• $M\begin{bmatrix}w_j\\\ldots\\w_{j+m - 1}\end{bmatrix}$ is the multiplication of a $d$ by $d*m$ matrix by a $d*m$ by $1$ vector, which results in a $d$ by $1$ vector. – Mark L. Stone May 13 '18 at 18:10
• Could you bit a bit more elaborate? Why is $M$ a $d$ by $d\times m$ matrix? The weights $m_{:,j}$ of the filters $d$ are just vectors right? So why is $M$ a 3 dimensional matrix if it is just composed of vectors? – Mr. President May 13 '18 at 19:02