Why is it hard to bring back a once “broken” neural network to usable results ? - Classification MNIST SGD

Question

I'm currently working through Michael Nielsens book "Neural Networks and Deep Learning" .

If I use his code/hyperparameters the network get's quite a good classification score. Even after the first epoch !

Output - Randomly initialized neural net

# 30 epochs, batch size = 10, learning rate = 3.0
net = network.Network([784, 20, 10])
net.SGD(training_data, 30, 10, 3.0, test_data=test_data)

Epoch 0: 8992 / 10000
Epoch 1: 9181 / 10000
Epoch 2: 9236 / 10000
Epoch 3: 9323 / 10000
Epoch 4: 9284 / 10000

I played around a bit with too high learning rates and as expected the classification results got worse.

Breaking a net with a high learning rate:

broken_net = network.Network([784, 20, 10])
# 30 epochs, batch size = 5000, learning rate = 300.0
broken_net.SGD(training_data, 30, 5000, 300.0, test_data=test_data)

Epoch 0: 3876 / 10000
Epoch 1: 3289 / 10000
Epoch 2: 3288 / 10000
Epoch 3: 3365 / 10000

Trying to bring it back to live:

Please note i use all the same hyperparameters as for the working example above.

# 30 epochs, batch size = 10, learning rate = 3.0
broken_net.SGD(training_data, 30, 10, 3.0, test_data=test_data)

Epoch 1: 3905 / 10000
Epoch 2: 3878 / 10000
Epoch 3: 3875 / 10000
Epoch 4: 3883 / 10000
Epoch 5: 3891 / 10000
Epoch 6: 3901 / 10000
Epoch 7: 3929 / 10000
Epoch 8: 3907 / 10000
Epoch 9: 3924 / 10000
..
..
Epoch 26: 4004 / 10000
Epoch 27: 3988 / 10000
Epoch 28: 3975 / 10000
Epoch 29: 3966 / 10000

Question

How can it be that a randomly initialized Neural Network gets accurate already after 1 epoch of training , but the broken one doesn't even get close to the performance of the randomly initialized net even after 30 epochs ?

What I would actually expect is that it's easier to get the "broken net" back to higher scores. Since the expectation value for digit classification would be 10% accuracy (for a random net). And the broken net had a accuracy of 33%.

I mean it could be that it trapped the gradient descent in a minimum but this phenomenon seems quite reproducible.

Answer 1

It's quite hard to say without knowing more details. It's likely that by updating with such a large learning rate, you've sent the weights off to some region where they are not amenable to further optimization.

If you're familiar with how important correct weight initialization is for training neural networks (indeed, with very careful initialization, it's possible to train 10000 layer networks!) -- then it shouldn't come as a surprise that by violently disturbing the weights, you may have messed up the initialization so much that it's not able to recover.