I have been dealing with a problem in his online text book:
L2 regularization sometimes automatically gives us something similar to the new approach to weight initialization. Suppose we are using the old approach to weight initialization. Sketch a heuristic argument that: supposing $\lambda$ is not too large, the weight decay will tail off when the weights are down to a size around $\frac{1}{\sqrt{n}}$, where $n$ is the total number of weights in the network.
My first idea to approach this problem was to compare weights before-and-after stochastic gradients. So here is the result:
Before:
- Each weight entry in $w^l$ is Gaussian distributed with mean 0 and var 1
- $w^l$ means the sum of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer
- $v^l$ means the variance of all the weights in $l^{th}$ layer from $(l-1)^{th}$ layer
- $w^2$: 75.94, $w^3$: 4.26, $|w^2|_1$: 33.03, $|w^3|_1$: 12.51, $v^2$: 1.01, $v^3$: 1.15
After:
- $w^2$: 130.23, $w^3$: -72.15, $|w^2|_1$: 12.75, $|w^3|_1$: 26.65, $v^2$: 0.08, $v^3$: 3.62
Weights in 2nd layer have been decreased, although the amounts of the decreases were not enough to hit $\frac{1}{\sqrt(n)}$. However, weights in 3rd layer increased when they were supposed to be lower the original. I'm curious if regularization does reduce weights. And is there any specific reason why weights in different layer are affected in the opposite way? What am I missing here? And is there any other way to approach the problem? Any insight please?
My codes are below (they ran on Colab):
#Download the dataset from Nielsen's github
!wget -L https://github.com/mnielsen/neural-networks-and-deep-learning/raw/master/data/mnist.pkl.gz
#Download mnist_loader.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/mnist_loader.py
#Download network2.py from Dobrzanski's github: his version's for python 3
!wget -L https://raw.githubusercontent.com/MichalDanielDobrzanski/DeepLearningPython/master/network2.py
import mnist_loader
import network2
import numpy as np
training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
net = network2.Network([784, 30, 10],cost=network2.CrossEntropyCost)
net.large_weight_initializer()
b0 = net.weights[0]
b1 = net.weights[1]
print(f' W^2 : {b0.sum():.2f}\t\t W^3 : {b1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(b0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(b1, ord=1):.2f}')
print(f' V^2 : {np.var(b0):.2f}\t\t V^3 : {np.var(b1):.2f}')
net.SGD(training_data, 30, 10, 0.1, lmbda = 5.0, evaluation_data=validation_data, monitor_evaluation_accuracy=True)
c0 = net.weights[0]
c1 = net.weights[1]
print(f' W^2 : {c0.sum():.2f}\t\t W^3 : {c1.sum():.2f}')
print(f'|W^2|: {np.linalg.norm(c0, ord=1):.2f}\t\t|W^3|: {np.linalg.norm(c1, ord=1):.2f}')
print(f' V^2 : {np.var(c0):.2f}\t\t V^3 : {np.var(c1):.2f}')
