How to propagate error from convolutional layer to previous layer？

Question

I've been trying to implement a simple convolutional neural network. But I stuck in some questions. To be specific, assume there are 3 layers in a convolutional pass, marked as l-1, l, l+1 -layer respectively. And the l-1 layer is the input layer with shape (num_channels, img_height, img_width),the l layer is the convolutional layer with kernel shape (num_kernels, num_channels, filter_height, filter_width), and the l+1 layer is the pooling layer with pooling size (pool_height, pool_width). In this setting, I encountered several problems in the back-propagation phase.

1. I've been stuck at this problem for over a week. I've studied many tutorials in neural networks, however, it seems that all of them stop with back-propagating the error to l layer with the formular: \begin{align} \delta_k^{(l)} = \text{upsample}\left((W_k^{(l)})^T \delta_k^{(l+1)}\right) \bullet f'(z_k^{(l)}) \end{align} The question is, how should I back-propagate error from the l layer to l-1 layer, in order to chain several conv-pooling passes together?

2. About up-sampling. Assume I take a simple mean pooling strategy with pooling size (2, 2). It is said the up-sampling is like:

\begin{matrix} \delta_{11}^{l+1} / \beta & \delta_{11}^{l+1} / \beta & \delta_{12}^{l+1} / \beta & \delta_{12}^{l+1} / \beta &...\\ \delta_{11}^{l+1} / \beta & \delta_{11}^{l+1} / \beta & \delta_{12}^{l+1} / \beta & \delta_{12}^{l+1} / \beta &...\\ \delta_{21}^{l+1} / \beta & \delta_{21}^{l+1} / \beta & \delta_{22}^{l+1} / \beta & \delta_{22}^{l+1} / \beta &...\\ \delta_{21}^{l+1} / \beta & \delta_{21}^{l+1} / \beta & \delta_{22}^{l+1} / \beta & \delta_{22}^{l+1} / \beta &...\\ ... &...&...&... \end{matrix} What confuses me is, what is the value of $\beta$, is it 1/4 or 1? What about L2 norm pooling?

If I take max pooling, it seems that I have to keep track of the position of the maximum value in each patch of feature maps of l layer, which is quite complicate to do. Am I wrong? Is there any better way?

Answer 1

Looks like the question is more about, how to propagate error through pooling layer.

You can find the answer here for max-pooling: https://datascience.stackexchange.com/questions/11699/backprop-through-max-pooling-layers and in general rule here in slide 11: http://www.slideshare.net/kuwajima/cnnbp

You are right is both account; during max pooling you have to maintain the index of max unit in forward propagation and is mean pooling, it is 1/m, m being in the size of pooling function, of the gradient.

Answer 2

During forward propagation, you need to remember the input for a standard dense layer, in order to calculate the gradient during the backprop.

Similarly for a pooling layer, you also need to remember which cell had the activation. The other cells in the pooling square do not backprop.

You can also think of it like how backprop for ReLU works. Since during input, you do Max(0, X_in), during backprop, the gradient doesn't backprop if X_in was not activated (or less than 0).