I have a FCNN with skip connection, consist of 7 layers of conv, maxpooling, Leaky Relu, BN in encoding path and 6 layers of conv, upsampling, Leaky Relu, BN in decoding path. Loss function is MSE and the optimizer SGD.
I removed 2 middle hidden layer (the deepest ones) and the training time for each epochs gets slower by a factor of 8!
Theoretically, I expect the shallower network to converge at later epochs compared to a deeper model. but decreased speed in single epochs seems to be counter intuitive for me, by reducing the number of hidden layers this means that network should learn fewer weights. Thus, I would expect an increased speed for each epoch.
Why this happens? Is this a known phenomena?