Temporal convolution for NLP I'm trying to follow Kalchbrenner et al. 2014 (http://nal.co/papers/Kalchbrenner_DCNN_ACL14) (and basically most of the papers in the last 2 years which applied CNNs to NLP tasks) and implement the CNN model they describe. Unfortunately, although getting the forward pass right, it seems like I have a problem with the gradients.
The convolution part of the forward pass looks like:
for samp_in in range(x.shape[0]): # every input sample
    for x_in in range(self.W.shape[0]):  # every input channel
        for k in range(self.W.shape[1]): # every output kernel
            for r in range(self.W.shape[2]): # every row in kernel/input
                self.output[samp_in, k, r, :] += np.convolve(x[samp_in, x_in, r, :], self.W[x_in, k, r, :], 'full')

            self.output[samp_in, k, :, :] += self.b[k]
            acti[samp_in, k, :, :] = self.output[samp_in, k, :, :]

While the backwards pass is:
# gradients wrt W, b
for x_in in range(self.input.shape[1]):
    for k in range(self.W.shape[1]):
        for r in range(prev_delta.shape[2]):
            self.Wgrad[x_in, k, r, :] = np.convolve(self.input[0, x_in, r, :], prev_delta[0, k, r, :], 'valid')

        self.bgrad[k] += np.sum(prev_delta[0, k, :, :])

# gradients wrt x
for x_in in range(self.input.shape[1]):
    for k in range(self.output.shape[1]):
        for r in range(prev_delta.shape[2]):
            delta[0, x_in, r, :] += np.convolve(prev_delta[0, k, r, :], self.W[x_in, k, r, :], 'valid')

This returns the correct size and dimensionality but the gradient checking is really off when connecting layers. Testing a single conv layer the results are correct, connecting 2 conv layers - also correct, but then, when adding MLP, Pooling, etc. it starts returning false gradients. All other types of layers were also tested separately and they are all correct. I'd therefore assume the problem starts with the calculation of the grad. wrt W_conv.
Does anyone have an idea or a useful link to a similar implementation?
 A: I'm working exactly on the same task now, implementing ConvNet for NLP ) Regarding reference implementations, there is a lot of general-purpose ConvNets out there, they are can be readily found with google. I think most of them don't support things like dynamic k-max pooling, most run only on GPU and some come as part of big machine learning libs. I know only one publicitly avaliale implementation of CNN tailored to NLP tasks (http://riejohnson.com/cnn_download.html), but it only runs  on GPU and uses somewhat different approach. 
To debug my own implementation I created a simplest possible syntetic task and simple convnet with one feature map (2x1) and one pooling layer, and then manually checked all problematic parts, step by step. After that I picked up more complex synthetic problem, added second feature map and so on, until whole thing seemed to work ok. It always good idea to start from simple cases.
It is also important to debug convolutional layer together with pooling layer, to ensure that backprop error correctly passed to convolutional layer. Otherwise system will fail to work, even when convolutional layer itself works fine.
I'm currently in process of testing it on sentiment classification task, again starting from simple architectures, to see whats really important and what is not. 
PS. 
its difficult to say anything more about your problem in general, many things are very implementation specific, but if you are interested, you can contact me directly so we can compare notes...
