# Drop-out as a multiplicative noise in deep neural networks

I am reading Ian Goodfellow's deep learning book, and I cannot understand the following lines:

Another important aspect of dropout is that the noise is multiplicative. If the noise were additive with fixed scale, then a rectified linear hidden unit with added noise could simply learn to have become very large in order to make the added noise insignificant by comparison. Multiplicative noise does not allow such a pathological solution to the noise robustness problem.

I dont understand what does it mean by "a rectified linear hidden unit with added noise could simply learn to have become very large in order to make the added noise insignificant by comparison. Multiplicative noise does not allow such a pathological solution to the noise robustness problem"?

• Hi, and welcome to Cross Validated! Nice question, but in the future it would be better to add a reference to the book paragraph (in this case, it's towards the end of paragraph 7.12), so that it's easier to read the full quote in context.. In addition to the reference, you may also add a link, but adding just the link without the reference is discouraged on this site because links can go stale. – DeltaIV Sep 2 '18 at 8:13

## 1 Answer

The correct answer is actually the opposite of the answer by DuttaA. Dropout works so well exactly because it doesn't allow the ReLU to make the noise insignificant by just making its output larger. Let's see why. To make things simpler, I'll use the same toy architecture used in the book, which is a 2-2-1 fully connected NN (figure 7.7 of the book): and I'll follow the same notation, thus $$h_i$$ is the activation of unit $$i$$, and $$\boldsymbol{\mu}=(\mu_1,\mu_2,\mu_3,\mu_4)$$ is the vector of dropout masks, i.e., it's a vector of binary values $$[0,1]$$. It's a 4-element vector because dropout is only applied to input or hidden units, but not to output units.

Now, how do we add noise to the neural vector using dropout? The noiseless output would be:

$$y = w_{21}h_1(w_{11}x_1+b_1)+w_{21}h_2(w_{12}x_2+b_2)+b_y$$

Let's consider additive noise: additive noise is usually added only to the input nodes. As a matter of fact, Ian explains in the paragraph just before the one you quoted that one of the advantages of dropout over traditional noise injection techniques is that the traditional methods only add noise at the input layer level, while dropout multiplies noise also at the hidden layers level. However, to simplify the treatment, I'll consider additive noise and dropout only at the hidden layer level (if you look around on GitHub, you'll find plenty of Keras models which don't use dropout for the input layer). Since the input layer is untouched, I'll make the following substitutions:

\begin{align} w_{11}x_1+b_1 &= a_1 \\ w_{12}x_2+b_2 &= a_2 \end{align}

Thus we have:

## Additive noise

$$y = w_{21}(h_1(a_1)+\epsilon_1)+w_{21}(h_2(a_2)+\epsilon_2)+b_y = \mathbf{w}_2(\mathbf{h}+\boldsymbol{\epsilon})+b_y$$

If $$\mathbf{h}$$ is made of unbounded activation functions (e.g., ReLU), the neural network can get rid of $$\boldsymbol{\epsilon}$$ just by making $$\mathbf{h}$$ larger and larger. In this simple example, this could be achieved simply by making $$\mathbf{w}_1$$ and/or $$\mathbf{b}$$ huge. This is a pathological solution to the noise problem! We would like the hidden units to learn to perform well, whether other hidden units are in the model or not, i.e., not to rely too much on the presence of all the other hidden units (this is one of the possible interpretations of dropout). Instead, we only told the neural network to use both the hidden units, make their outputs huge and then of course reduce it so that loss is small (i.e., that the predicted $$\hat{y}_j$$ are close to the correct $$y_j$$ on the training set), by making $$\mathbf{w}_2$$ small. This is a classic "pathological solution" found by an optimizer - similar to when we use an optimizer for zero finding (i.e., we look for solutions which make the objective function, or loss, 0), and it finds it by making two additive terms huge, one going to $$-\infty$$ and the other going to $$+\infty$$.

Let's see how this is different with multiplicative noise.

## Multiplicative noise (dropout)

$$y = w_{21}(h_1(a_1)\mu_1)+w_{21}(h_2(a_2)\mu_2)+b_y = \mathbf{w}_2(\mathbf{h}\odot \boldsymbol{\mu})+b_y$$

Now, since $$\mu_1$$ and $$\mu_1$$ are independent and identically distributed Bernoulli random variables (i.e., their value is either 0 or 1 with a certain probability $$p_{drop}$$), the NN can't just learn to make $$\mathbf{h}$$ huge in order to get rid of the noise. If $$\mu_1$$ is randomly 0, then whatever the (finite) value of $$h_1$$, its contribution to $$y$$ during those training steps is exactly 0. If the NN wants to find anyway a solution which yields a decent value of $$y$$, even in the presence of this pesky multiplicative noise, it will have to learn the fact that it cannot always rely on $$h_1$$ and $$h_2$$ both being in the network 1.

PS this part of the book is great! I've never seen dropout explained so well, and the similarities with bagging (as well as the differences with boosting) so clearly underlined.

1of course, dropout would never be applied to such a small network, unless the function to be learned was extremely simple and the training set very small. With just 2 hidden units, if we drop each units randomly and independently of the other, the neural network has very little capacity to learn anything. Just think that if $$p_{drop}=0.5$$ (a common choice for hidden units), then in 1 out of every 4 training steps both hidden units will be 0, and during that step the error won't backpropagate at all.

• thanks delta, your explanation is quite phenomenal! Indeed this book is great – samra irshad Sep 4 '18 at 4:21
• @samrairshad glad you liked it! If you think it's a good answer you can upvote it (I'm not sure you have enough reputation to do that, actually). BTW, compliments! Someone who starts her journey in Deep Learning from Ian's book, instead than from one of the thousand aimless "Deep Learning in 1 day" tutorials on YouTube, definitely is worthy of praise. You'll still need to use tutorials (or more applied books) to actually produce something, but with that kind of background you'll have an edge on other practitioners. – DeltaIV Sep 4 '18 at 7:52
• @DeltaIV nice..my bad..i did not check the definition – DuttaA Sep 4 '18 at 11:02