# L2 regularization with standard weight initialization

Suppose we have a feedforward neural network with L2 regularization and we train it using SGD initializing the weights with the standard Gaussian. The weight update scheme can be written as:

$$w \rightarrow \left( 1 - {\eta \lambda \over n} \right)w - {\eta \over m} \sum_x{\partial C_x \over \partial w}$$

where $w$ is any given weight in the network, $\eta$ is the learning rate, $\lambda$ is the regularization rate, $m$ is the size of the mini-batch, the sum is over all the training examples $x$ in a given mini-batch and $C_x$ is the cost function for a given training example $x$.

I want to understand four characteristics of this configuration:

1) Supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay.

2) Provided $\eta \lambda ≪ n$ the weights will decay by a factor of $e^{-{\eta \lambda \over m}}$ per epoch.

3) Supposing $\lambda$ is not too large, the weight decay will tail off when the weights are down to a size around ${1 \over n_w}$, where $n_w$ is the total number of weights in the network.

4) How does this relate to weight initialization using $N(0, {1 \over n_{in}})$, where $n_{in}$ is the number of inputs to a neuron (no regularization)?

1) With standard initialization of weights, during the fist epochs of learning, we will often have ${1 \over m} \sum_x{\partial C_x \over \partial w} \approx 0$ and weight decay will be dominant.

2) We could replace $\eta \lambda ≪ n$ with $\lambda$ being a constant and $n \to \infty$. Then we have weight decay at $\lim\limits_{n \to \infty}\left( 1 - {\eta \lambda \over n} \right)^{n \over m}=e^{-{\eta \lambda \over m}}$ per epoch.

3) I don't quite see how we can tie the tapering of the decay weights to $w \approx {1 \over n_w}$. It would seem that the weight decay component vanishes in later epochs, irrespectively of the actual level of the weights (which doesn't have to be $0$ thanks to the gradient component).

4) Perhaps the relation is that L2 cuts the weights and in doing so emulates the effects of $N(0, {1 \over n_{in}})$ initiaization.

I would appreciate any comments on the correctness of 1) and 2), and any ideas on what is going on in 3) and 4).

• Would you mind to define all symbols that you are using? Nov 8 '17 at 14:31
• @KarelMacek edited Nov 8 '17 at 22:26

1. I agree with your explanation. I would only add the reason why each $$\cfrac{\partial C_x}{\partial w}\approx0$$, which I believe is due to the saturation of the neurons mantained at the first epochs given the ''big'' standard deviation from the random weights generator ($$\sigma=1$$).
3. and 4) Yes, I think you got it right. I believe so due to the fact that if we want the weights of our network to be normally disributed by $$\mathcal{N}\left(0,\sigma \approx \cfrac{1}{\sqrt{n}}\right)$$ (following the reasoning of the book at pages 94-96) we shoud be able to reduce its initial $$\sigma = 1$$ from the initial Normal distribution $$\mathcal{N}(0, 1)$$ (given by the random weights generator). This is something we can heuristically do by re-scaling them with a factor $$\frac{1}{\sqrt{n}}$$.