So as an experiment, I trained different instances of the LeNet network from scratch on MNIST, with some modification for each instance. By "from scratch", I mean that before each training session, I randomly initialised all layers of the LeNet network.
Note: Freezing a layer (used below) means that the frozen layer does not update its weights via backprop, rather retains its randomly initialised weights throughout the session.
I designed and trained the network in Python using TensorFlow, in case you think that's relevant.
Training Session 1: Randomly initialised all layers of LeNet network N1. Trained N1 on the MNIST dataset until convergence. Got 99+% validation accuracy.
Training Session 2: Randomly initialised all layers of LeNet network N2, and froze the first layer. Trained N2 on the MNIST dataset until convergence. Got 99+% validation accuracy.
Training Session 3: Randomly initialised all layers of LeNet network N3, and froze the second layer. Trained N3 on the MNIST dataset until convergence. Got 99+% validation accuracy.
You get the idea, this goes on for all the layers in the network.
I made a plot of accuracy versus epochs for all the training sessions (the plots are on another machine, I'll try to upload them here sometime). I expected that the networks with one frozen layer (N2, N3, ...) would perhaps converge to a lower accuracy; or would converge onto the same accuracy as the unfrozen network (N1) at a slower rate. I expected this because I believed that the networks with a frozen layer would have lesser modelling power (due to fewer trainable parameters) as compared to an unfrozen network.
However, the plots showed that the accuracies of all the different networks (N1, N2, N3, ...) converged to approximately the same value and at the same rate. That is, the maximum value of accuracy obtained by each network was the same, and the rise in the accuracy relative to the epochs was also the same.
I am not able to make sense of this seemingly counter-intuitive result. Does anybody have any explanation for this? Or perhaps my expectation of some other results was based on incorrect arguments?