3
$\begingroup$

The textbook I am reading just says that dropout tends to increase the total training time by 2~3 times but does not explain why. Can someone provide an intuitive explanation?

$\endgroup$
2
  • $\begingroup$ Does this refer to any particular type of neural network, convolutional or recurrent, for example? $\endgroup$
    – Dave
    Apr 20, 2020 at 21:25
  • $\begingroup$ I've also noticed this behavior ~ are you reading deep learning with python by any chance?~ The time/epoch is going from 10 to 120 seconds =( This seems a bit extraordinary to me. $\endgroup$ May 13, 2020 at 13:49

1 Answer 1

3
$\begingroup$

Suppose all the weights of a network are fixed at some value. If you mask some of these weights, the loss will likely be different than if you mask a different set of these weights. This is what dropout is doing: it's an additional source of randomness. This means that the estimation of the loss is noisy, so the estimation of the gradient is noisy, so the optimizer will move in directions which are influenced by this noise.

This increases training time compared to a network trained without dropout because the to find a local minimum because sometimes the noise will cause the optimizer to move away from a local minimum instead of towards it. When this happens, the optimizer must make additional steps to move back in the correct direction. These additional steps require more iterations, and therefore more training time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.