I have been dealing with a problem in his online text book:

L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization. Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that: supposing $\lambda$ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where $n$ is the total number of weights in the network.

My first idea to approach this problem was to compare weights before-and-after stochastic gradients. So here is the result:


  • Each weight entry in $w^l$ is Gaussian distributed with mean 0 and var 1
  • $w^l$ means the sum of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer
  • $v^l$ means the variance of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer
  • $w^2$: 75.94, $w^3$: 4.26, $|w^2|_1$: 33.03, $|w^3|_1$: 12.51, $v^2$: 1.01, $v^3$: 1.15


  • $w^2$: 130.23, $w^3$: -72.15, $|w^2|_1$: 12.75, $|w^3|_1$: 26.65, $v^2$: 0.08, $v^3$: 3.62

Weights in 2nd layer have been decreased, although the amounts of the decreases were not enough to hit $\frac{1}{\sqrt(n)}$. However, weights in 3rd layer increased when they were supposed to be lower the original. I'm curious if regularization does reduce weights. And is there any specific reason why weights in different layer are affected in the opposite way? What am I missing here? And is there any other way to approach the problem? Any insight please?

My codes are below (they ran on Colab):

#Download the dataset from Nielsen's github
!wget -L https://github.com/mnielsen/neural-networks-and-deep-learning/raw/master/data/mnist.pkl.gz

#Download mnist_loader.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/mnist_loader.py

#Download network2.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/network2.py
import mnist_loader
import network2
import numpy as np

training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
net = network2.Network([784, 30, 10],cost=network2.CrossEntropyCost)
b0 = net.weights[0]
b1 = net.weights[1]
print(f' W^2 : {b0.sum():.2f}\t\t W^3 : {b1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(b0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(b1, ord=1):.2f}')
print(f' V^2 : {np.var(b0):.2f}\t\t V^3 : {np.var(b1):.2f}')
net.SGD(training_data, 30, 10, 0.1, lmbda = 5.0, evaluation_data=validation_data, monitor_evaluation_accuracy=True)
c0 = net.weights[0]
c1 = net.weights[1]
print(f' W^2 : {c0.sum():.2f}\t\t W^3 : {c1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(c0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(c1, ord=1):.2f}')
print(f' V^2 : {np.var(c0):.2f}\t\t V^3 : {np.var(c1):.2f}')
  • 1
    $\begingroup$ Very closely related: stats.stackexchange.com/questions/206178. Also see stats.stackexchange.com/a/452468/919 for an example where regularization first decreases the magnitude of a coefficient in a model and then increases it. $\endgroup$
    – whuber
    Commented Jul 22, 2021 at 21:14
  • $\begingroup$ Thank you for your comment, Whuber. I will check them up once I go back home tonight. $\endgroup$
    – Kay
    Commented Jul 23, 2021 at 2:14

1 Answer 1


When there are local optima, it's quite possible to end up in larger weights due to the randomness of the procedure and the heuristics used (e.g. SGD). But, considering the global optima in both cases, adding regularization decreases the weights' norms in which they are regularized for.

Let the total loss be decomposed as the following, where $L_e$ is the reconstruction error, like cross-entropy, or MSE/MAE; and $L_r$ be the regularization error, e.g. $L_2$ regularization (squared sum of the weights).

$$L=L_e + \lambda L_r$$

When there is no regularization, weights are chosen such that $L_e$ is minimized. Let the $L_r$ term be $A$ with these weights. So, the regularized loss would be $L_{e,min} + \lambda A$. Let the solution weights be $w^*$.

If we had used the regularized loss from the beginning, $L_e$ would be larger (or at least the same) since no other $w$ can achieve less magnitude. And, $L_r$ should be lower as well because a higher magnitude means a sub-optimal solution since $w^*$ has a lower absolute magnitude.

So, if you're able to reach global optima, it's certain that the regularization will decrease the collective magnitude of the weights. However, individual layers can be affected differently as what matters is the total.

Regarding the sum of weights w/o taking the absolute value, you can't guarantee anything because the sum could be close to zero but the weights can be enormously large. So, I would always check the weights' magnitudes in the form they're regularized, e.g. absolute value if $L_1$ regularization is applied, or squared sum if $L_2$ regularization is applied etc.

Regarding the $1/\sqrt n$ tail off cut-off, I haven't seen the statement before or checked the corresponding section in the book you mentioned. This answer merely tries to address your title question in a more general manner, and warns about the validity of some of the metrics you checked.

  • $\begingroup$ Thank you for your insight, Gunes. I have one thing unclear. The goal of the regularization of the network is to avoid overfitting. If $L_e$ increases and $\lambda L_r$ is a positive number, then $L$ would increase compared to the original $L$ without regularization. But we generally want to lower $L$ in order to achieve better accuracy of the model. So here, are we kind of sacrificing accuracy in order to avoid overfitting? $\endgroup$
    – Kay
    Commented Jul 23, 2021 at 2:11
  • $\begingroup$ We aim better performance in test data, not the training. So, regularization is used for increasing the generalization performance, not the training. $\endgroup$
    – gunes
    Commented Jul 23, 2021 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.