2
$\begingroup$

I have a FCNN with skip connection, consist of 7 layers of conv, maxpooling, Leaky Relu, BN in encoding path and 6 layers of conv, upsampling, Leaky Relu, BN in decoding path. Loss function is MSE and the optimizer SGD.

I removed 2 middle hidden layer (the deepest ones) and the training time for each epochs gets slower by a factor of 8!

Theoretically, I expect the shallower network to converge at later epochs compared to a deeper model. but decreased speed in single epochs seems to be counter intuitive for me, by reducing the number of hidden layers this means that network should learn fewer weights. Thus, I would expect an increased speed for each epoch.

Why this happens? Is this a known phenomena?

$\endgroup$
2
$\begingroup$
  • With different network architecture, you may need different hyperparameters (e.g. learning rate, or batch size), the convergence speed will depend on those as well.
  • Another hypothesis might be that the bigger network was more flexible because of having more parameters, so it was easier for it to adapt to the data. Smaller network offers a much more constrained class of models that it could fit, so it might have harder time with finding the appropriate configuration of the parameters. Of course, this is just a hypothesis.
$\endgroup$
5
  • 1
    $\begingroup$ right but this would explain why the network converges at later epochs, say instead of 75th at 150th but it doesn't explain why single epochs takes longer times, after all fewer weights to compute should have been resulted in a decrease in computational effort for each epoch... $\endgroup$
    – Farnaz
    Jun 26 '20 at 10:05
  • $\begingroup$ @Farnaz "convergence speed" is defined in terms of epochs, it you ask "why code runs slower" it is a matter of implementational details & neither would be on-topic in here, not is answerable without looking at your code and implementational decisions made in the software you're using. $\endgroup$
    – Tim
    Jun 26 '20 at 10:10
  • $\begingroup$ No, I am not talking about why the code is slower, I exactly mean times of epochs which increased after removing middle layers... $\endgroup$
    – Farnaz
    Jun 26 '20 at 10:52
  • $\begingroup$ @Farnaz probably some kind of low level optimizations in the code work better between both scenarios, but this you should rather ask on some kind of support site for the software you're using. $\endgroup$
    – Tim
    Jun 26 '20 at 10:58
  • 2
    $\begingroup$ My guess is that you have a bug somewhere. I would print out the time taken to do the forward/backward pass over each layer, and see what changes when you remove the layers. $\endgroup$ Jun 26 '20 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.