How does training affect the norm of weight matrices?

Question

I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint.

My loss function is the classic $L(W) = \frac{1}{2}\sum_{i \in [n]} || F(W,x_i) - y_i||^2$.

This means that every weight matrix $W \in \mathbb{R}^{m \times m}$. These matrices are started with random initialisation $W_{i,j} = \mathcal{N}(0, \frac{2}{m})$. First of all, what's the Frobenius norm of this matrix?

If it was symmetric, Wigner's theory would suggest us that

$$\mathbb{E}||W||_F = 2\sqrt{m}*2/\sqrt{m} = 4$$

I'm not sure it is right, but it should. What I need is an answer to the following questions:

Once I start training, how do the weights changes affect the norm of the weight matrices?

What's the expected norm change after every iteration?

What's the expected norm of a well trained network?

(I have never specified if I need the $||*||_2$ norm or the Frobenius one because I actually need both, so whatever answer with one of them is valid :D )

Thank you!

Answer 1

You ask an interesting question, but unfortunately you consider a case which is far removed from mainstream research:

• you consider vector regression, instead than multiclass classification (if you're actually interested in multiclass classification, then you wrote the wrong loss function)
• you don't use any regularization: no batch normalization, weight decay or SGD, and no learning rate decay. Actually, I'm surprised your neural network can learn anything at all! What are the targets?

I cannot give a complete answer, but I can at least give you some indications. First of all, let's correct a couple errors:

• Not all the weights matrices are $W^{l} \in \mathbb{R}^{m \times m}$. When $l \in \{1, L\}$, you have resp. $W^1 \in \mathbb{R}^{m \times d}$ and $W^L \in \mathbb{R}^{k \times m}$

• $\mathbb{E}||W||_F \neq 2\sqrt{m}*2/\sqrt{m} = 4$ It's the Euclidean operator norm (which is a proper induced norm) that satisfies this identity, not the Frobenius norm. See, e.g., https://arxiv.org/pdf/1608.06953.pdf. For the Frobenius norm and $l\not\in \{1,L\}$ I get:

  $$ \mathbb{E}||W^l||_F =\sqrt{\sum_i^m\sum^m_j\mathbb{E}[w_{ij}^2]}=\sqrt{m^2\frac{4}{m^2}}=2 $$

  not sure why you're getting another result.

Having said that, recent research suggests that part of the success in training modern neural networks is due to implicit control of the weight norm. If you're using batch norm and weight decay, then one can prove that weight decay, controlling the weight norm, prevents the effective step size to decrease, which would hinder optimization (see Hoffer et al., "Norm matters: efficient and accurate normalization schemes in deep networks", 2018). In your case, since you're not using weight decay or batch norm, this isn't true. However, the change in weight norm does depend on the learning rate $\eta$ and on the initialization. As a matter of fact,

$$W^l_{t+1}=W^l_t-\eta\frac{\partial\ell}{\partial W^l_t}(W^1_t,\dots,W^L_t)$$

Thus, it's quite obvious that, depending on $\eta$, the change in $W^l_t$ and thus in its norm will be different. Without knowing anything about $\eta$, I don't think much can be said, but you could have a look at the tools of nonlinear random matrix theory for deep learning:

Jeffrey Pennington, Pratik Worah, Nonlinear random matrix theory for deep learning, 2017

Answer 2

I am not really sure what the goal of your questions is. How the weights change when training depend on the optimization algorithm you employ (SGD, BFGS, CG,...), the parameters for optimization, and the data you are using. Now, once you understand that, what is the use of knowing what is the exact expected norm after each iteration (other than verifying the accuracy of your implementation)?

In such a setting I guess you are also using some form of regularization. It would be more interesting to see how different forms of regularization help you with the problem, and what kind of features (weights) the network learns. Or maybe also, how that regularization affects the update rules of your weights during training.

What do you learn from knowing the concrete value of the norm of the weights? What motivates you to make those questions?.