I am reading Deep Learning by Goodfellow, Bengio, and Courville. On Page 238, they introduce noises as one way to do regularization. Quote:

Noise applied to the weights can also be interpreted as equivalent (under some assumptions) to a more traditional form of regularization, encouraging stability of the function to be learned. Consider the regression setting, where we wish to train a function $\hat{y}(\mathbf{x})$ that maps a set of features to a scalar using the least-squares cost function between the model predictions $\hat{y}(\textbf{x})$ and the true values $y$:

\begin{equation}J = \mathbb{E}_{p(x,y)}[(\hat{y}(\mathbf{x})-y)^2]\end{equation}

The training set consists of $m$ labeled examples $\{(\mathbf{x}^{(1)},y^{(1)}), \cdots , (\mathbf{x}^{(m)},y^{(m)})\}$. We now assume that with each input presentation we also include a random perturbation $\varepsilon_{\mathbf{W}} \sim \mathcal{N}(\mathbf{\varepsilon};\mathbf{0},\eta\mathbf{I})$ of the network weights. Let us imagine that we have a standard $l$-layer MLP. We denote the perturbed model as $\hat{y}_{\mathbf{\varepsilon}_{\mathbf{W} }}(\mathbf{x})$. Despite the injection of noise, we are still interested in minimizing the squared error of the output of the network. The objective function thus becomes

$$\tilde{J}_{\mathbf{W}} =\mathbb{E}_{p(x,y,\varepsilon_{\mathbb{W}})}[(\hat{y}_{\varepsilon_{\mathbf{W}}}(\mathbf{x})-y)^2] = \mathbf{E}_{p(\mathbf{x},y,\mathbf{\varepsilon}_{\mathbf{W}})}\left[\hat{y}_{\varepsilon_{\mathbf{W}}}(\mathbf{x})-2y\hat{y}_{\varepsilon_\mathbf{W}}+y^2\right]$$

For small $\eta$, the minimization of $J$ with added weight noise (with covariance $\eta \mathbf{I}$) is equivalent to minimization of $J$ with an additional regularization term $\eta \operatorname{\mathbb E}_{p(\textbf{x},y)}[\|\nabla_{\mathbf{W}} \hat{y}(\textbf{x})\|^2]$.

I have two questions about this passage.

(1) What is the point of the equation $$\tilde{J}_{\mathbf{W}} =\mathbb{E}_{p(x,y,\varepsilon_{\mathbb{W}})}[(\hat{y}_{\varepsilon_{\mathbf{W}}}(\mathbf{x})-y)^2]=\mathbf{E}_{p(\mathbf{x},y,\mathbf{\varepsilon}_{\mathbf{W}})}\left[\hat{y}_{\varepsilon_{\mathbf{W}}}(\mathbf{x})-2y\hat{y}_{\varepsilon_\mathbf{W}}+y^2\right] \text{?}$$ What are they trying to show?

(2) Where does $\eta \operatorname{\mathbb E}_{p(\textbf{x},y)}[\|\nabla_{\mathbf{W} }\hat{y}(\textbf{x})\|^2]$ come from? How do I derive this?


1 Answer 1


Equation (1) is just the expected loss function with randomly perturbed weights. The whole idea is to show that $J$ with a regularization is equivalent to $\tilde J$ under certain conditions. (You have typo there, the last expression should have the square of the first term).

There is some notational complexity in the book. The neural network is also a function of $W$ as well as $\mathbf x$. If we consider $x$ and $y$ as constants for a moment just for brevity, we can write the following using Taylor series:

$$\hat y_\epsilon(W)=\hat y(W+\epsilon)\approx \hat y(W)+\nabla\hat y(W)^T\epsilon$$

This is a first-order approximation and it's a good one when $\epsilon$ is small. If we just substitute this into (1), we get the following:

$$\tilde J=\mathbb E\left[(\hat y - y)^2 - 2 (\hat y - y) (\nabla\hat y)^T \epsilon + \|\nabla \hat y^T\ \epsilon\|^2\right]=\underbrace{\mathbb E[(y-\hat y)^2]}_J+\mathbb E\left[\|\nabla\hat y^T\epsilon\|^2\right]$$

Also note that $\mathbb E[\epsilon]=0$ and independent of other variables, thus the middle term cancels. The second term is basically, $\mathbb E[\|\nabla \hat y\|^2]\operatorname{Var}(\epsilon)=\eta E[\|\nabla \hat y\|^2]$ due to the complete independence of each $\epsilon_i$.


  • The Taylor expansion assumes denominator layout in the vector differentiation without loss of generality.
  • The expectation notation does not strictly need the probability distributions, therefore the simplified notation.
  • Notation of $\hat y$, $\epsilon$ and partial derivative wrt $W$ is also simplified for brevity.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.