The cause to the issue was a programming error. I had a wrong variable assignment. To make this answer still useful, I'll document how I found out.
Firstly, I got an existing implementation to be able to compare your network's performance. Well-known frameworks such as Tensoflow or Pytorch are recommended, but in my case, I stuck to the example code of the book.
Then I needed to have the same starting conditions. Set a fixed seed on one implementation, generate the network there and import it into the other. As always, testing it with a single neuron first will prevent unneeded effort, I found the issue at this single neuron already. I set the seed on the book's implementation (Numpy's
random.seed()) and generated the network. When you have just one neuron, you can just
print the weight and bias.
Then import this network into the other framework. My implementation had such a functionality, but in another data format. With only one neuron, you can just copy-paste the values. Make sure to use the same activation function in both networks.
Finally, get training data. If you test on a single neuron, one or two samples may be enough to find the error. I just used a
0.0 -> 0.0 example. It would be useful if you don't shuffle the set in one implementation in a way that can't be reproduced by the other.
Then you can start testing. Run the reference network and compare it to your implementation. Is the output the same as in your implementation? Do one backpropagation. Are all values the same? In my case they weren't. If they are, you at least know that a single neuron works as intended, try to test with bigger networks. In my case the values after the backpropagation differed. I proceeded to log every single value that made sense until I found where the difference came from and found the error.
In my case, it sort of worked like this:
- Run the network, same output: Feedforward maths ✓.
- Run the backpropagation, difference of bias differs: backprop fails ✗.
- Log cost function prime, same: Cost function ✓.
- Log cost function prime * activation function prime: same ✓.
- Realize that the difference of the bias is just the cost function prime, not the product of both.
- Disbelief, check out the code nevertheless.
- Find the bug:
// The derivative of the cost with respect to the bias is just the
// same as the derivative with respect to the sum (the wieght * input
// part is just added).
- let d_cost_d_bias = d_cost_d_output;
+ let d_cost_d_bias = d_cost_d_sum;
(the comment even says so lol.)
Finally, you can do the same procedure again on both networks, to validate that the fix actually fixed the problem. After testing it on the single neuron, I also ran the entire network again, to verify that the network's performance lines up roughly with the reference implementation's performance.
If these kinds of tests for made with every commit, that'd make them unit tests or integration test respectively. But who needs that right?
I hope that you, dear future reader, can find and fix your bugs!