Is this drop in training accuracy due to a statistical or programming error?

Question

I am training a Feedforward NN on the MNIST dataset and hit a roadblock. My accuracy only goes to about 80% and the cost only about 0.32. This is the case for NNs that have a hidden layer with 10 neurons. Some tests with other example code (Michael Nielsen's Neural Networks and Deep Learning book) let me conclude that my network has too few nodes to sufficiently learn the classification (I could barely achieve 90% accuracy with the same model and his example code).

However, as soon as I add one neuron, the training peaks at about 90% accuracy and then goes downhill. I suspected overfitting, so I logged the accuracy for the training set beside the accuracy for the validation set. According to this post, I had expected a growing discrepancy between those values, but turns out, both peaked and the decreased: (the X-Axis is in number of mini-batches / 100)

I am running with a batch size of 10. The learning rates I've tried are 0.3, 0.1 and 0.02, all yielding the same result, just slower.

As the network is a self made implementation, the cause might also be a bug, but I don't think because of the working example with 10 hidden neurons.

What might also cause this drop? Is this overfitting? Or are there indicies to other mistakes?

---

For anyone interested, you can view the code here (Rust). The entire PRNG is seed-based, so all scenarios are reproducable. To run it, just cargo run --release (in the network/ dir). Try to change the network dimensions and the learn factor in main.rs.

Answer 1

The cause to the issue was a programming error. I had a wrong variable assignment. To make this answer still useful, I'll document how I found out.

---

Firstly, I got an existing implementation to be able to compare your network's performance. Well-known frameworks such as Tensoflow or Pytorch are recommended, but in my case, I stuck to the example code of the book.

Then I needed to have the same starting conditions. Set a fixed seed on one implementation, generate the network there and import it into the other. As always, testing it with a single neuron first will prevent unneeded effort, I found the issue at this single neuron already. I set the seed on the book's implementation (Numpy's random.seed()) and generated the network. When you have just one neuron, you can just print the weight and bias.

Then import this network into the other framework. My implementation had such a functionality, but in another data format. With only one neuron, you can just copy-paste the values. Make sure to use the same activation function in both networks.

Finally, get training data. If you test on a single neuron, one or two samples may be enough to find the error. I just used a 0.0 -> 0.0 example. It would be useful if you don't shuffle the set in one implementation in a way that can't be reproduced by the other.

Then you can start testing. Run the reference network and compare it to your implementation. Is the output the same as in your implementation? Do one backpropagation. Are all values the same? In my case they weren't. If they are, you at least know that a single neuron works as intended, try to test with bigger networks. In my case the values after the backpropagation differed. I proceeded to log every single value that made sense until I found where the difference came from and found the error.

In my case, it sort of worked like this:

• Run the network, same output: Feedforward maths ✓.
• Run the backpropagation, difference of bias differs: backprop fails ✗.
• Log cost function prime, same: Cost function ✓.
• Log cost function prime * activation function prime: same ✓.
• Realize that the difference of the bias is just the cost function prime, not the product of both.
• Disbelief, check out the code nevertheless.
• Find the bug:

         // The derivative of the cost with respect to the bias is just the
         // same as the derivative with respect to the sum (the wieght * input
         // part is just added).
-        let d_cost_d_bias = d_cost_d_output;
+        let d_cost_d_bias = d_cost_d_sum;

(the comment even says so lol.)

Finally, you can do the same procedure again on both networks, to validate that the fix actually fixed the problem. After testing it on the single neuron, I also ran the entire network again, to verify that the network's performance lines up roughly with the reference implementation's performance.

If these kinds of tests for made with every commit, that'd make them unit tests or integration test respectively. But who needs that right?

I hope that you, dear future reader, can find and fix your bugs!