# How does training affect the norm of weight matrices?

I have a neural network $$F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$$ with $$L$$ layers, $$m$$ neurons per layer, ReLu activation, softmax on the last layer and $$n$$ datapoint.

My loss function is the classic $$L(W) = \frac{1}{2}\sum_{i \in [n]} || F(W,x_i) - y_i||^2$$.

This means that every weight matrix $$W \in \mathbb{R}^{m \times m}$$. These matrices are started with random initialisation $$W_{i,j} = \mathcal{N}(0, \frac{2}{m})$$. First of all, what's the Frobenius norm of this matrix?

If it was symmetric, Wigner's theory would suggest us that

$$\mathbb{E}||W||_F = 2\sqrt{m}*2/\sqrt{m} = 4$$

I'm not sure it is right, but it should. What I need is an answer to the following questions:

Once I start training, how do the weights changes affect the norm of the weight matrices?

What's the expected norm change after every iteration?

What's the expected norm of a well trained network?

(I have never specified if I need the $$||*||_2$$ norm or the Frobenius one because I actually need both, so whatever answer with one of them is valid :D )

Thank you!

• How the heck it's possible that your training procedure converges at all? You don't use any regularization (SGD, BatchNorm or weight decay) and you don't have any skip connections. How many layers do you use? Unless $L=1$ or $L=2$, I would expect such a neural network to either learn slowly, or plateau to a large value of $\ell$. Probably it depends on your learning problem. You say that $X$ is a random subset of $S^d=\{\mathbf{x}\in \mathbb{R}^d\mid \vert\vert \mathbf{x} \vert\vert \le 1\}$. What are the targets, i.e., $y_i$? – DeltaIV Jan 20 at 15:59
• Even more surprising, it sounds like you don't use any learning rate decay. I have to say, this really sounds like a prank :-) can you please show a learning curve? To be fair, learning rate decay is strictly necessary only if you use SGD with constant minibatch size. Still, I think the fact that your NN can be trained, is quite noteworthy. Evidently, the problem to be learned is very simple. – DeltaIV Jan 20 at 16:13
• The problem is incredibly simple: 1-2 thousands of points randomly generated on the unit sphere. The problem is that, even if the class I'm trying to learn is this simple, I still can't find an answer to my question. I can't image tackling the problem with a real dataset because, as you mentioned in your answer, there would be to many variables to take care of. The targets $y_i$ are random generated as well, and they are the classic output vectors $(1,0,\dots,0), (0,1,\dots,0) \dots$. – Alfred Jan 20 at 22:39
• But again, my goal is to understand how weight change really works, and starting from a very easy NN looks like the only way to face the problem. – Alfred Jan 20 at 22:39
• Wait. I don't understand. First of all, if the targets are $(1,0,\dots,0),(0,1,\dots,0)\dots$, is this actually vector regression, or aren't you thinking of multiclass classification? Second, if I understand correctly, there's no relationship at all between features and labels (i.e., targets), meaning that the accuracy on a test set would be essentially that of random choice ($\frac{1}{k}$ since you have $k$ classes). This isn't a very meaningful problem: your NN cannot learn any pattern, thus it's essentially trying to memorize the train set. It's hard to conclude anything at this point! – DeltaIV Jan 20 at 23:10

You ask an interesting question, but unfortunately you consider a case which is far removed from mainstream research:

• you consider vector regression, instead than multiclass classification (if you're actually interested in multiclass classification, then you wrote the wrong loss function)
• you don't use any regularization: no batch normalization, weight decay or SGD, and no learning rate decay. Actually, I'm surprised your neural network can learn anything at all! What are the targets?

I cannot give a complete answer, but I can at least give you some indications. First of all, let's correct a couple errors:

• Not all the weights matrices are $$W^{l} \in \mathbb{R}^{m \times m}$$. When $$l \in \{1, L\}$$, you have resp. $$W^1 \in \mathbb{R}^{m \times d}$$ and $$W^L \in \mathbb{R}^{k \times m}$$

• $$\mathbb{E}||W||_F \neq 2\sqrt{m}*2/\sqrt{m} = 4$$ It's the Euclidean operator norm (which is a proper induced norm) that satisfies this identity, not the Frobenius norm. See, e.g., https://arxiv.org/pdf/1608.06953.pdf. For the Frobenius norm and $$l\not\in \{1,L\}$$ I get:

$$\mathbb{E}||W^l||_F =\sqrt{\sum_i^m\sum^m_j\mathbb{E}[w_{ij}^2]}=\sqrt{m^2\frac{4}{m^2}}=2$$

not sure why you're getting another result.

Having said that, recent research suggests that part of the success in training modern neural networks is due to implicit control of the weight norm. If you're using batch norm and weight decay, then one can prove that weight decay, controlling the weight norm, prevents the effective step size to decrease, which would hinder optimization (see Hoffer et al., "Norm matters: efficient and accurate normalization schemes in deep networks", 2018). In your case, since you're not using weight decay or batch norm, this isn't true. However, the change in weight norm does depend on the learning rate $$\eta$$ and on the initialization. As a matter of fact,

$$W^l_{t+1}=W^l_t-\eta\frac{\partial\ell}{\partial W^l_t}(W^1_t,\dots,W^L_t)$$

Thus, it's quite obvious that, depending on $$\eta$$, the change in $$W^l_t$$ and thus in its norm will be different. Without knowing anything about $$\eta$$, I don't think much can be said, but you could have a look at the tools of nonlinear random matrix theory for deep learning:

Jeffrey Pennington, Pratik Worah, Nonlinear random matrix theory for deep learning, 2017

• Thank you very much for the references and the answer (Yes, the size of the first and last weight matrix was a distraction mistake :D ) – Alfred Jan 20 at 22:41

I am not really sure what the goal of your questions is. How the weights change when training depend on the optimization algorithm you employ (SGD, BFGS, CG,...), the parameters for optimization, and the data you are using. Now, once you understand that, what is the use of knowing what is the exact expected norm after each iteration (other than verifying the accuracy of your implementation)?

In such a setting I guess you are also using some form of regularization. It would be more interesting to see how different forms of regularization help you with the problem, and what kind of features (weights) the network learns. Or maybe also, how that regularization affects the update rules of your weights during training.

What do you learn from knowing the concrete value of the norm of the weights? What motivates you to make those questions?.

• You are right, I should have added more details. The algorithm I'm using is gradient descend, the step size $\eta$ is arbitrary (it shouldn't change the norm of the weights much. The data I'm using are random points taken from the sphere $S^d$. No regularization, simple feedforward neural network with ReLu activation. What motivates me to do make those questions is that I need to compute the Jacobian of a neural network, I don't need it for any practical application. – Alfred Jan 19 at 21:19
• If just want to do that, give pytorch a try. It is quite straightforward to get the jacobian for each layer, and see how the weights change. – jpmuc Jan 19 at 22:05