# Is this drop in training accuracy due to a statistical or programming error?

I am training a Feedforward NN on the MNIST dataset and hit a roadblock. My accuracy only goes to about 80% and the cost only about 0.32. This is the case for NNs that have a hidden layer with 10 neurons. Some tests with other example code (Michael Nielsen's Neural Networks and Deep Learning book) let me conclude that my network has too few nodes to sufficiently learn the classification (I could barely achieve 90% accuracy with the same model and his example code).

However, as soon as I add one neuron, the training peaks at about 90% accuracy and then goes downhill. I suspected overfitting, so I logged the accuracy for the training set beside the accuracy for the validation set. According to this post, I had expected a growing discrepancy between those values, but turns out, both peaked and the decreased: (the X-Axis is in number of mini-batches / 100)

I am running with a batch size of 10. The learning rates I've tried are 0.3, 0.1 and 0.02, all yielding the same result, just slower.

As the network is a self made implementation, the cause might also be a bug, but I don't think because of the working example with 10 hidden neurons.

What might also cause this drop? Is this overfitting? Or are there indicies to other mistakes?

For anyone interested, you can view the code here (Rust). The entire PRNG is seed-based, so all scenarios are reproducable. To run it, just cargo run --release (in the network/ dir). Try to change the network dimensions and the learn factor in main.rs.

• An example of catastrophic forgetting perhaps? – Frans Rodenburg Jun 2 at 19:08
• You write "As the network is a self made implementation, the cause might also be a bug" Can you replicate this behavior if you make an identical network in a standard neural network library such as Tensorlfow or Pytorch? – Sycorax Jun 2 at 19:17
• That information doesn't really do much to demonstrate that there are not errors in your implementation. It's all well and good to have code that seems to work well in one specific case, but I am wary whenever novel code demonstrates strange sensitivity like this. In my experience, this can arise if there are small, subtle bugs that only become apparent when the configuration is changed. Perhaps there's an indexing error or something similar? That's the kind of problem that might not manifest in every situation, but is only obvious in some. – Sycorax Jun 2 at 20:05
• The only real way to validate whether your implementation is correct is to test whether it matches a “gold standard.” Tensorflow and Pytorch aren’t infallible, but they’ve got a lot of people checking their work. It will also let you compare every intermediate step, which is nice because NNs have a lot of pieces, and doing 9 of them correctly can still have problems if the 10th one is bugged. – Sycorax Jun 2 at 20:13
• – kjetil b halvorsen Jun 3 at 15:30

The cause to the issue was a programming error. I had a wrong variable assignment. To make this answer still useful, I'll document how I found out.

Firstly, I got an existing implementation to be able to compare your network's performance. Well-known frameworks such as Tensoflow or Pytorch are recommended, but in my case, I stuck to the example code of the book.

Then I needed to have the same starting conditions. Set a fixed seed on one implementation, generate the network there and import it into the other. As always, testing it with a single neuron first will prevent unneeded effort, I found the issue at this single neuron already. I set the seed on the book's implementation (Numpy's random.seed()) and generated the network. When you have just one neuron, you can just print the weight and bias.

Then import this network into the other framework. My implementation had such a functionality, but in another data format. With only one neuron, you can just copy-paste the values. Make sure to use the same activation function in both networks.

Finally, get training data. If you test on a single neuron, one or two samples may be enough to find the error. I just used a 0.0 -> 0.0 example. It would be useful if you don't shuffle the set in one implementation in a way that can't be reproduced by the other.

Then you can start testing. Run the reference network and compare it to your implementation. Is the output the same as in your implementation? Do one backpropagation. Are all values the same? In my case they weren't. If they are, you at least know that a single neuron works as intended, try to test with bigger networks. In my case the values after the backpropagation differed. I proceeded to log every single value that made sense until I found where the difference came from and found the error.

In my case, it sort of worked like this:

• Run the network, same output: Feedforward maths ✓.
• Run the backpropagation, difference of bias differs: backprop fails ✗.
• Log cost function prime, same: Cost function ✓.
• Log cost function prime * activation function prime: same ✓.
• Realize that the difference of the bias is just the cost function prime, not the product of both.
• Disbelief, check out the code nevertheless.
• Find the bug:
         // The derivative of the cost with respect to the bias is just the
// same as the derivative with respect to the sum (the wieght * input