I'm currently working through Michael Nielsens book "Neural Networks and Deep Learning" .
If I use his code/hyperparameters the network get's quite a good classification score. Even after the first epoch !
Output - Randomly initialized neural net
# 30 epochs, batch size = 10, learning rate = 3.0
net = network.Network([784, 20, 10])
net.SGD(training_data, 30, 10, 3.0, test_data=test_data)
Epoch 0: 8992 / 10000
Epoch 1: 9181 / 10000
Epoch 2: 9236 / 10000
Epoch 3: 9323 / 10000
Epoch 4: 9284 / 10000
I played around a bit with too high learning rates and as expected the classification results got worse.
Breaking a net with a high learning rate:
broken_net = network.Network([784, 20, 10])
# 30 epochs, batch size = 5000, learning rate = 300.0
broken_net.SGD(training_data, 30, 5000, 300.0, test_data=test_data)
Epoch 0: 3876 / 10000
Epoch 1: 3289 / 10000
Epoch 2: 3288 / 10000
Epoch 3: 3365 / 10000
Trying to bring it back to live:
Please note i use all the same hyperparameters as for the working example above.
# 30 epochs, batch size = 10, learning rate = 3.0
broken_net.SGD(training_data, 30, 10, 3.0, test_data=test_data)
Epoch 1: 3905 / 10000
Epoch 2: 3878 / 10000
Epoch 3: 3875 / 10000
Epoch 4: 3883 / 10000
Epoch 5: 3891 / 10000
Epoch 6: 3901 / 10000
Epoch 7: 3929 / 10000
Epoch 8: 3907 / 10000
Epoch 9: 3924 / 10000
..
..
Epoch 26: 4004 / 10000
Epoch 27: 3988 / 10000
Epoch 28: 3975 / 10000
Epoch 29: 3966 / 10000
Question
How can it be that a randomly initialized Neural Network gets accurate already after 1 epoch of training , but the broken one doesn't even get close to the performance of the randomly initialized net even after 30 epochs ?
What I would actually expect is that it's easier to get the "broken net" back to higher scores. Since the expectation value for digit classification would be 10% accuracy (for a random net). And the broken net had a accuracy of 33%.
I mean it could be that it trapped the gradient descent in a minimum but this phenomenon seems quite reproducible.