1
$\begingroup$

I read the original paper on Dropout (Srivastava, et. al.) and it seems to suggest that even dense networks with lots of parameters can be trained in such a way that they don't overfit to the training samples i.e. by using dropout. Now, I have trained two convolutional nets (for a personal project) with depths 2 and 3 respectively. The one with depth 2 is generalizing better (on unseen samples) even though I am using dropout (0.5) in both the cases. If the theory on dropout mentioned in the original paper is correct then I should be getting roughly the same test error with deeper network too because the size of data is huge (> 200K samples). The network with 3 convolutional layers should perform at least close to or better than the one with only 2 convolutional layers. Why is it not happening?

$\endgroup$
1
  • 1
    $\begingroup$ There does not appear to be any information provided to support an objective answer. Please consult our help center for guidance on asking an answerable question. $\endgroup$ – whuber Sep 28 '16 at 16:05
3
$\begingroup$

Dropout is a regularization technique, which makes it more difficult to overfit. However, it does not make it impossible to overfit, especially if you have a large number of parameters in your convolutional layers. So you could still be experiencing overfitting in your 3 layer model.

Another potential problem with dropout is slow convergence. In the 2014 Srivastava paper on Dropout, they mention that they needed to drastically increase the learning rate of their dropout networks (and use weight normalization) to get the networks to converge. Dropout modifies the variance of the signal at each layer, which must be taken into account by either normalization or weight initialization to ensure good gradient propagation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.