1
$\begingroup$

I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer.

Please note that the whole thing was built from scratch following the neuralnetworksanddeeplearning book and other resources, so I'm aware there are ready-to-use libraries out there, as well as more optimized algorithms to perform certain operations (like the FFT convolution algorithm), but the point of this thing was to understand how a network works and to implement everything without using 3rd party libraries.

I'm quite sure the fully connected layers work fine, as I've tried training a network with the MNIST dataset and it quickly gets to over 96% in less than 60 epochs, just like the one in the book mentioned above.

I went on and implemented the convolutional layers, and they seem to be working fine too, in fact if I try to have a network like this one (sigmoid activation on all the layers):

  • Convolution: 28*28*1 inputs, 20 5*5*1 kernels
  • Fully Connected: 24*24*20 inputs, 100 outputs
  • Fully Connected: 100 inputs, 10 outputs, cross-entropy cost

And here's what I got: enter image description here

As you can see, the network quickly reaches over 96% accuracy in just 3 epochs.

But, as soon as I try to put a 2*2 pooling layer between the convolutional and fully connected layer (just like Michael Nielsen did in the 6th chapter of his book), here's what I get:

enter image description here

The network reaches 30% and then just gets stuck there.

Update - here's a plot of the cross-entropy cost using the pooling layer:

enter image description here

The cost gets stuck there at around 2.52, and the accuracy stays at around 30%, as stated before. I am aware that using a max pooling layer causes only the weights that contributed to the activated neurons to be updated, but the thing is that here I'm exactly replicating the network structure used by Michael Nielsen in his book, and he states that this network easily gets to over 98% of accuracy in less than 60 epochs (using the same training parameters). Here instead it's clear that my network will never even get close to 98%, even after a 100 epochs.

Update #2 - here's another cost plot over 200 epochs:

enter image description here

I've checked and everything seems to be working fine (but of course I can be wrong, and I probably am, since this thing is not working as it should), the forward pooling layer takes the maximum value in each 2*2 window across all the input images (doing so for each depth layer), and during the backpropagation through the pooling layer I just upscale the input delta using the previous outputs from the convolutional layer, so that each delta goes to the pixel that had the maximum value in the convolutional output, while all the other pixels get 0.

Where should I start looking for the error? I mean, clearly something happens with the pooling layer, but the code looks fine to me, both in theory and with the actual Unit tests I've added to the project. What's the right approach to take here to investigate the issue? Are there probable explanations for this issue?

Here's the code I'm using, if anyone has .NET Core 2.x installed and wants to try the library out (The code to download/parse the MNIST dataset is already in the library):

((float[,] X, float[,] Y) training, (float[,] X, float[,] Y) test) = DataParser.LoadDatasets();
INeuralNetwork network = NetworkTrainer.NewNetwork(
    NetworkLayers.Convolutional((28, 28, 1), (5, 5), 20, ActivationFunctionType.Identity),
    NetworkLayers.Pooling((24, 24, 20), ActivationFunctionType.Sigmoid),
    NetworkLayers.FullyConnected(12 * 12 * 20, 100, ActivationFunctionType.Sigmoid),
    NetworkLayers.FullyConnected(100, 10, ActivationFunctionType.Sigmoid, CostFunctionType.CrossEntropy));
await NetworkTrainer.TrainNetworkAsync(network, (training.X, training.Y), 60, 10, null,
    new TestParameters(test, new Progress<BackpropagationProgressEventArgs>(p =>
    {
        Printf($"Epoch {p.Iteration}, cost: {p.Cost}, accuracy: {p.Accuracy}");
    })));

Thanks for your help!

$\endgroup$

closed as off-topic by DeltaIV, Michael Chernick, jbowman, Sycorax, Stephan Kolassa Nov 30 '17 at 8:39

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – DeltaIV, Michael Chernick, jbowman, Sycorax, Stephan Kolassa
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Accuracy is not helpful for diagnosing these types of problems. Plot the cross-entropy loss as a function of time. I have no idea what's going on in your code, but a common pitfall is that people neglect that weights are only updated for the neurons that were active in the max layer. $\endgroup$ – Sycorax Nov 29 '17 at 20:17
  • $\begingroup$ @Sycorax Thanks for your reply, I've updated the question with a plot of the cross-entropy cost! $\endgroup$ – Sergio0694 Nov 29 '17 at 21:01
  • $\begingroup$ It seems like the network is still decreasing until epoch 5. Does the cross-entropy loss stay flat after epoch 6? (The accuracy plots extend to 40 epochs.) $\endgroup$ – Sycorax Nov 29 '17 at 21:02
  • $\begingroup$ @Sycorax Yup, it keeps hovering around that value. I'm doing another run right now with a larger batch size (just to take less time, now at epoch 19), will post that chart as well as soon as it's done, if it helps. I have no clue where the error in my code might be. Thank you! $\endgroup$ – Sergio0694 Nov 29 '17 at 21:15
  • $\begingroup$ It looks like the loss is continuing to improve, just slowly. Playing with learning rate and other optimization parameters might be necessary. $\endgroup$ – Sycorax Nov 30 '17 at 5:34
1
$\begingroup$

Build a unit test for your max-pooling layer.

Here's an example: Forwards: https://github.com/hughperkins/DeepCL/blob/master/test/testpoolingforward.cpp#L20

PoolingForward *poolingForward = PoolingForward::instanceForTest( cl, false, numPlanes, imageSize, poolingSize );
float data[] = { 1, 2, 5, 3,
                 3, 8, 4, 1,
                 3, 33, 14,23,
                 -1, -3.5f,37.4f,5
};
int outputNumElements = poolingForward->getOutputNumElements( batchSize );
int *selectors = new int[outputNumElements];
float *output = new float[outputNumElements];

poolingForward->forward( batchSize, data, selectors, output );

EXPECT_EQ( selectors[0], 3 );
EXPECT_EQ( selectors[1], 0 );
EXPECT_EQ( selectors[2], 1 );
EXPECT_EQ( selectors[3], 2 );

EXPECT_EQ( output[0], 8 );
EXPECT_EQ( output[1], 5 );
EXPECT_EQ( output[2], 33 );
EXPECT_EQ( output[3], 37.4f );

Backwards: https://github.com/hughperkins/DeepCL/blob/master/test/testpoolingbackward.cpp#L19

PoolingBackward *poolingBackprop = PoolingBackward::instanceForTest( cl, false, numPlanes, imageSize, poolingSize );
float errors[] = {
    3, 5,
    2, 9
};
int selectors[] = {
    2, 1,
    0, 3
};
float *errorsForUpstream = new float[ poolingBackprop->getInputNumElements( batchSize ) ];

poolingBackprop->backward( batchSize, errors, selectors, errorsForUpstream );

float expectedErrorsForUpstream[] = {
    0,0,0,5,
    3,0,0,0,
    2,0,0,0,
    0,0,0,9,
};
for( int i = 0; i < 16; i++ ) {
    ASSERT_EQ( expectedErrorsForUpstream[i], errorsForUpstream[i] );
}
$\endgroup$
  • $\begingroup$ Hello, I have a ton of Unit tests in the library, and the pool layer seems to be working fine (I even tried to replicate these two tests). See the Unit tests here: github.com/Sergio0694/NeuralNetwork.NET/blob/master/Unit/… Thanks again! $\endgroup$ – Sergio0694 Nov 29 '17 at 23:17
  • $\begingroup$ Ok. So, one possibility is you forgot to zero your arrays, and this only gets noticed at runtime. Another possibility is that there's an error elsewhere in your library, that only manifests itself in the presence of the pooling layer. Something you can do if you havent already done it is use finite-difference differentiation to check other layers. You can also fix weights, and compare the results with eg convnetjs, which is simple, easy to read, and thus likely correct. (In my experience, pooling layer is relatively simple, bug-free; conv layers tend to be the buggiest...) $\endgroup$ – Hugh Perkins Nov 30 '17 at 10:16
  • $\begingroup$ Thanks for your reply! So, not zeroing arrays is not the case here, since I allocate a new array every time and C# being a managed language automatically zeroes them upon creation. The thing that baffles me is that fully connected layers works fine (even when using 2 or more of them), and using only conv + fc seems to be working fine too, as I quickly reach over 96% accuracy. I mean, if the pool layer is fine and the conv isn't, wouldn't that prevent the conv + fc test from training so well? I'll try to use manual weights and double check though, will let you know! $\endgroup$ – Sergio0694 Nov 30 '17 at 16:19
  • 1
    $\begingroup$ Well, I double checked the code of every layer and it turns out the pooling was indeed fine, the issue was with the convolutional output. Basically I'm executing the activation after the pooling to increase the performance (to have fewer values to process), so when it's followed by a pool layer, the convolutional layer just copies its outputs (identity activation basically). I messed up the number of bytes to copy and so I was losing 3/4ths of outputs for every convolutional layer. Stupid error, but everything's working fine at last, thanks! $\endgroup$ – Sergio0694 Nov 30 '17 at 19:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.