I'm reading this paper on a convolutional neural network for modelling sentences, and I'm having some trouble understanding section $3.5$. Please consider the following text:

We denote a feature map of the $i$-th order by $F^i$. As in convolutional networks for object recognition, to increase the number of learnt feature detectors of a certain order, multiple feature maps $F_1^i,\ldots, F_n^i$ may be computed in parellel at the same layer. Each feature map $F_j^i$ is computed by convolving a distinct set of filters arranged in a matrix $m_{j,k}^i$ with each feature map $F_k^{i-1}$ of the lower order $i-1$ and summing the results: $$F_j^i = \sum_{k=1}^n m_{j,k}^i*F_k^{i-1}$$ where $*$ indicates the wide convolution. The weights $m_{j,k}^i$ form an order-$4$-tensor. After the wide convolution, first dynamic $k$-max pooling and then the non-linear function are applied individually to each map.

I've tried to figure out why the matrices of weights $m_{j,k}^i$ form an order-$4$-tensor (even though there are only three indices), by figuring out the feature maps:

For the first layer we have:

$$F_1^1 = \sum_{k=1}^1m_{1,k}^1 * F_k^0 = m_{1,1}^1*F_1^0\\ F_2^1 =\sum_{k=1}^1m_{2,k}^1 * F_k^0 = m_{2,1}^1*F_1^0$$ I think we just sum to one here since the input for the first layer is just the sentence matrix $s$ (so $F_1^0 = s$). For the second layer we have: $$F_1^2 = \sum_{k=1}^2m_{1,k}^2*F_k^1 = m_{1,1}^2*F_1^1 + m_{1,2}^2*F_2^1\\ F_2^2 = \sum_{k=1}^2m_{2,k}^2*F_k^1 = m_{2,1}^2*F_1^1 + m_{2,2}^2*F_2^1$$ If this is correct that means that we need six different matrices for the two layers combined: $$m_{1,1}^1, m_{2,1}^1, m_{1,1}^2, m_{1,2}^2, m_{2,1}^2, m_{2,2}^2$$ Question: Why/how do these six matrices form a order-$4$-tensor?

The only thing I can think of is that it has something to do with the fact that the word embeddings in this paper have dimension $d = 4$ (there are four different filters), and that there is a different matrix $m_{j,k}^i$ for each of those four dimensions. This doesn't make sense though, since the matrices are used on a two-dimensional input, where one of those two dimensions is the dimension of the filter. In order words: the fact that the $m_{j,k}^i$ are matrices means that every row already corresponds to a filter dimension.


Under any reasonable definition, you are right three indices would mean an order 3 tensor. In fact I would call the $m^i_{j,k}$ a set of two dimensional tensors (i.e. matrices) indexed by i. The point is that the i index is an index that takes finitely many integer values. The 'j' and 'k' components are different as they index a set of real numbers. However, you will have to check the paper to see how he is doing things. As Gertrude Stein once said, a definition is a definition is a definition.

| cite | improve this answer | |
  • $\begingroup$ oops- I truly meant to leave my 'answer' as a comment and not an answer ! $\endgroup$ – meh May 14 '18 at 14:10
  • $\begingroup$ Yeah well this is the only point in the paper where this supposed tensor is mentioned unfortunately. In the rest of the paper just matrices and vertices are just (that all make sense btw). $\endgroup$ – Mr. President May 15 '18 at 7:19
  • $\begingroup$ In which case regardles of whether it's a bizarre definition or whether it's a typo, it's not important. I have actually lectured and written notes on back propagation and it's clear to me that there is no sensible way in which the matrices involved can be thought to be 4 dimensional. $\endgroup$ – meh May 15 '18 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.