2
$\begingroup$

I have a FCNN with skip connection, consist of 7 layers of conv, maxpooling, Leaky Relu, BN in encoding path and 6 layers of conv, upsampling, Leaky Relu, BN in decoding path. Loss function is MSE and the optimizer SGD.

I removed 2 middle hidden layer (the deepest ones) and the training time for each epochs gets slower by a factor of 8!

Theoretically, I expect the shallower network to converge at later epochs compared to a deeper model. but decreased speed in single epochs seems to be counter intuitive for me, by reducing the number of hidden layers this means that network should learn fewer weights. Thus, I would expect an increased speed for each epoch.

Why this happens? Is this a known phenomena?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$
  • With different network architecture, you may need different hyperparameters (e.g. learning rate, or batch size), the convergence speed will depend on those as well.
  • Another hypothesis might be that the bigger network was more flexible because of having more parameters, so it was easier for it to adapt to the data. Smaller network offers a much more constrained class of models that it could fit, so it might have harder time with finding the appropriate configuration of the parameters. Of course, this is just a hypothesis.
Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ right but this would explain why the network converges at later epochs, say instead of 75th at 150th but it doesn't explain why single epochs takes longer times, after all fewer weights to compute should have been resulted in a decrease in computational effort for each epoch... $\endgroup$
    – Farnaz
    Jun 26, 2020 at 10:05
  • $\begingroup$ @Farnaz "convergence speed" is defined in terms of epochs, it you ask "why code runs slower" it is a matter of implementational details & neither would be on-topic in here, not is answerable without looking at your code and implementational decisions made in the software you're using. $\endgroup$
    – Tim ♦
    Jun 26, 2020 at 10:10
  • $\begingroup$ No, I am not talking about why the code is slower, I exactly mean times of epochs which increased after removing middle layers... $\endgroup$
    – Farnaz
    Jun 26, 2020 at 10:52
  • $\begingroup$ @Farnaz probably some kind of low level optimizations in the code work better between both scenarios, but this you should rather ask on some kind of support site for the software you're using. $\endgroup$
    – Tim ♦
    Jun 26, 2020 at 10:58
  • 2
    $\begingroup$ My guess is that you have a bug somewhere. I would print out the time taken to do the forward/backward pass over each layer, and see what changes when you remove the layers. $\endgroup$
    – Simon Alford
    Jun 26, 2020 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.