Why regularization does not always reduce weights? (Michael Nielson's book) I have been dealing with a problem in his online text book:

L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization. Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that: supposing $\lambda$ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where $n$ is the total number of weights in the network.

My first idea to approach this problem was to compare weights before-and-after stochastic gradients. So here is the result:
Before:


*

*Each weight entry in $w^l$ is Gaussian distributed with mean 0 and var 1

*$w^l$ means the sum of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer

*$v^l$ means the variance of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer

*$w^2$: 75.94,   $w^3$: 4.26,   $|w^2|_1$: 33.03,  $|w^3|_1$: 12.51, $v^2$: 1.01,    $v^3$: 1.15


After:


*

*$w^2$: 130.23,  $w^3$: -72.15, $|w^2|_1$: 12.75,  $|w^3|_1$: 26.65, $v^2$: 0.08,    $v^3$: 3.62


Weights in 2nd layer have been decreased, although the amounts of the decreases were not enough to hit $\frac{1}{\sqrt(n)}$. However, weights in 3rd layer increased when they were supposed to be lower the original. I'm curious if regularization does reduce weights. And is there any specific reason why weights in different layer are affected in the opposite way? What am I missing here? And is there any other way to approach the problem? Any insight please?
My codes are below (they ran on Colab):
#Download the dataset from Nielsen's github
!wget -L https://github.com/mnielsen/neural-networks-and-deep-learning/raw/master/data/mnist.pkl.gz

#Download mnist_loader.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/mnist_loader.py

#Download network2.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/network2.py
import mnist_loader
import network2
import numpy as np

training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
net = network2.Network([784, 30, 10],cost=network2.CrossEntropyCost)
net.large_weight_initializer()
b0 = net.weights[0]
b1 = net.weights[1]
print(f' W^2 : {b0.sum():.2f}\t\t W^3 : {b1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(b0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(b1, ord=1):.2f}')
print(f' V^2 : {np.var(b0):.2f}\t\t V^3 : {np.var(b1):.2f}')
net.SGD(training_data, 30, 10, 0.1, lmbda = 5.0, evaluation_data=validation_data, monitor_evaluation_accuracy=True)
c0 = net.weights[0]
c1 = net.weights[1]
print(f' W^2 : {c0.sum():.2f}\t\t W^3 : {c1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(c0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(c1, ord=1):.2f}')
print(f' V^2 : {np.var(c0):.2f}\t\t V^3 : {np.var(c1):.2f}')

 A: When there are local optima, it's quite possible to end up in larger weights due to the randomness of the procedure and the heuristics used (e.g. SGD). But, considering the global optima in both cases, adding regularization decreases the weights' norms in which they are regularized for.
Let the total loss be decomposed as the following, where $L_e$ is the reconstruction error, like cross-entropy, or MSE/MAE; and $L_r$ be the regularization error, e.g. $L_2$ regularization (squared sum of the weights).
$$L=L_e + \lambda L_r$$
When there is no regularization, weights are chosen such that $L_e$ is minimized. Let the $L_r$ term be $A$ with these weights. So, the regularized loss would be $L_{e,min} + \lambda A$. Let the solution weights be $w^*$.
If we had used the regularized loss from the beginning, $L_e$ would be larger (or at least the same) since no other $w$ can achieve less magnitude. And, $L_r$ should be lower as well because a higher magnitude means a sub-optimal solution since $w^*$ has a lower absolute magnitude.
So, if you're able to reach global optima, it's certain that the regularization will decrease the collective magnitude of the weights. However, individual layers can be affected differently as what matters is the total.
Regarding the sum of weights w/o taking the absolute value, you can't guarantee anything because the sum could be close to zero but the weights can be enormously large. So, I would always check the weights' magnitudes in the form they're regularized, e.g. absolute value if $L_1$ regularization is applied, or squared sum if $L_2$ regularization is applied etc.
Regarding the $1/\sqrt n$ tail off cut-off, I haven't seen the statement before or checked the corresponding section in the book you mentioned. This answer merely tries to address your title question in a more general manner, and warns about the validity of some of the metrics you checked.
