0
$\begingroup$

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as follows: let's say we decide on a $0.5$ probability of dropping any particular node from the first hidden layer onwards. Let's also assume that we're using linear activation for now with no bias.

The activations in the second hidden layer will be linear functions of activations in the first hidden layer and if the dropout probability is $0.5$, the expected value of the activation of any node in the second hidden layer is half of the "actual" magnitude (by "actual" magnitude, I mean the magnitude of the activation of the node in the full (no dropout) neural network).

Again, since the nodes in the second hidden layer have $0.5$ probability of being dropped out, the expected magnitude of the third hidden layer activations will be $0.5\times 0.5$ of the "actual" magnitude, and so on. This effect gets carried forward through the layers all the way to the output layer, the expected magnitude dropping by a factor of $0.5$ in each layer.

Thus the final output activation value and hence $\hat y$ will be grossly underestimated and the loss will be overestimated. Thus we need to scale up the activation values of each layer before performing a forward propagation step.

Is this correct or have I missed something or got something completely wrong? Also, are there any references that give a rigorous mathematical explanation of the need for scaling up in dropout, and why dropout has a regularizing effect?

Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.