I'm trying to follow Kalchbrenner et al. 2014 (http://nal.co/papers/Kalchbrenner_DCNN_ACL14) (and basically most of the papers in the last 2 years which applied CNNs to NLP tasks) and implement the CNN model they describe. Unfortunately, although getting the forward pass right, it seems like I have a problem with the gradients.

The convolution part of the forward pass looks like:

for samp_in in range(x.shape[0]): # every input sample
    for x_in in range(self.W.shape[0]):  # every input channel
        for k in range(self.W.shape[1]): # every output kernel
            for r in range(self.W.shape[2]): # every row in kernel/input
                self.output[samp_in, k, r, :] += np.convolve(x[samp_in, x_in, r, :], self.W[x_in, k, r, :], 'full')

            self.output[samp_in, k, :, :] += self.b[k]
            acti[samp_in, k, :, :] = self.output[samp_in, k, :, :]

While the backwards pass is:

# gradients wrt W, b
for x_in in range(self.input.shape[1]):
    for k in range(self.W.shape[1]):
        for r in range(prev_delta.shape[2]):
            self.Wgrad[x_in, k, r, :] = np.convolve(self.input[0, x_in, r, :], prev_delta[0, k, r, :], 'valid')

        self.bgrad[k] += np.sum(prev_delta[0, k, :, :])

# gradients wrt x
for x_in in range(self.input.shape[1]):
    for k in range(self.output.shape[1]):
        for r in range(prev_delta.shape[2]):
            delta[0, x_in, r, :] += np.convolve(prev_delta[0, k, r, :], self.W[x_in, k, r, :], 'valid')

This returns the correct size and dimensionality but the gradient checking is really off when connecting layers. Testing a single conv layer the results are correct, connecting 2 conv layers - also correct, but then, when adding MLP, Pooling, etc. it starts returning false gradients. All other types of layers were also tested separately and they are all correct. I'd therefore assume the problem starts with the calculation of the grad. wrt W_conv.

Does anyone have an idea or a useful link to a similar implementation?


closed as unclear what you're asking by Andy, Xi'an, Nick Cox, John, Glen_b Mar 1 '15 at 23:36

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ You might want to consider Theano, which does automatic differentiation and frees you from the headache of getting derivates right. $\endgroup$ – bayerj Mar 1 '15 at 21:23
  • $\begingroup$ Thanks for the comment, this kind of convolution is a lot of pain in Theano. It's a row-wise 1 dimensional full convolution which Theano doesn't seem to like very much. On top of that Theano doesn't like k-max pooling either. Combining the two with the fact that by doing it all myself I get to learn much more, I decided to set a side all the common frameworks and do it all meself. Thanks though $\endgroup$ – Ido_f Mar 1 '15 at 22:30

I'm working exactly on the same task now, implementing ConvNet for NLP ) Regarding reference implementations, there is a lot of general-purpose ConvNets out there, they are can be readily found with google. I think most of them don't support things like dynamic k-max pooling, most run only on GPU and some come as part of big machine learning libs. I know only one publicitly avaliale implementation of CNN tailored to NLP tasks (http://riejohnson.com/cnn_download.html), but it only runs on GPU and uses somewhat different approach.

To debug my own implementation I created a simplest possible syntetic task and simple convnet with one feature map (2x1) and one pooling layer, and then manually checked all problematic parts, step by step. After that I picked up more complex synthetic problem, added second feature map and so on, until whole thing seemed to work ok. It always good idea to start from simple cases.

It is also important to debug convolutional layer together with pooling layer, to ensure that backprop error correctly passed to convolutional layer. Otherwise system will fail to work, even when convolutional layer itself works fine.

I'm currently in process of testing it on sentiment classification task, again starting from simple architectures, to see whats really important and what is not.


its difficult to say anything more about your problem in general, many things are very implementation specific, but if you are interested, you can contact me directly so we can compare notes...

  • $\begingroup$ Thanks for the comment. They do seems to implement a different approach, which, as you said, is not very helpful in our case. I wanted to contact you, but your public profile does not have any contact details except for your LinkedIn page which, since we don't have any mutual connections, I also can't see. Would you send me a mail? $\endgroup$ – Ido_f Mar 1 '15 at 20:26
  • $\begingroup$ What did you end up doing? $\endgroup$ – Morteza Shahriari Nia Dec 1 '16 at 19:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.