Training LeNet on MNIST with frozen layers

Question

So as an experiment, I trained different instances of the LeNet network from scratch on MNIST, with some modification for each instance. By "from scratch", I mean that before each training session, I randomly initialised all layers of the LeNet network.

Note: Freezing a layer (used below) means that the frozen layer does not update its weights via backprop, rather retains its randomly initialised weights throughout the session.

I designed and trained the network in Python using TensorFlow, in case you think that's relevant.

1. Training Session 1: Randomly initialised all layers of LeNet network N1. Trained N1 on the MNIST dataset until convergence. Got 99+% validation accuracy.

2. Training Session 2: Randomly initialised all layers of LeNet network N2, and froze the first layer. Trained N2 on the MNIST dataset until convergence. Got 99+% validation accuracy.

3. Training Session 3: Randomly initialised all layers of LeNet network N3, and froze the second layer. Trained N3 on the MNIST dataset until convergence. Got 99+% validation accuracy.

You get the idea, this goes on for all the layers in the network.

I made a plot of accuracy versus epochs for all the training sessions (the plots are on another machine, I'll try to upload them here sometime). I expected that the networks with one frozen layer (N2, N3, ...) would perhaps converge to a lower accuracy; or would converge onto the same accuracy as the unfrozen network (N1) at a slower rate. I expected this because I believed that the networks with a frozen layer would have lesser modelling power (due to fewer trainable parameters) as compared to an unfrozen network.

However, the plots showed that the accuracies of all the different networks (N1, N2, N3, ...) converged to approximately the same value and at the same rate. That is, the maximum value of accuracy obtained by each network was the same, and the rise in the accuracy relative to the epochs was also the same.

I am not able to make sense of this seemingly counter-intuitive result. Does anybody have any explanation for this? Or perhaps my expectation of some other results was based on incorrect arguments?

Answer 1

Freezing a layer of a neural network during training is equivalent to having a random nonlinear projection in your algorithm. Surprisingly, perhaps, this can work quite well (though contentious, look up the ELM [1]) in practice. So the most likely answer is that it really isn't doing anything to your convergence rate since the transform happens to preserve sufficient information that it makes no difference.

This may also be interesting for you to test. Have you tried it with multiple frozen layers, ideally right after each other? Or with different initializing distributions? I would imagine that would make a difference -- if your transform throws away sufficient information then it wouldn't be possible to get the same accuracy.

With respect to the idea of loss of modelling power, that depends somewhat on how large the network you're training is. However, generally speaking, modern architectures are far more than expressive enough to classify MNIST.

[https://en.wikipedia.org/wiki/Extreme_learning_machine]