I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint.
My loss function is the classic $L(W) = \frac{1}{2}\sum_{i \in [n]} || F(W,x_i) - y_i||^2$.
This means that every weight matrix $W \in \mathbb{R}^{m \times m}$. These matrices are started with random initialisation $W_{i,j} = \mathcal{N}(0, \frac{2}{m})$. First of all, what's the Frobenius norm of this matrix?
If it was symmetric, Wigner's theory would suggest us that
$$\mathbb{E}||W||_F = 2\sqrt{m}*2/\sqrt{m} = 4$$
I'm not sure it is right, but it should. What I need is an answer to the following questions:
Once I start training, how do the weights changes affect the norm of the weight matrices?
What's the expected norm change after every iteration?
What's the expected norm of a well trained network?
(I have never specified if I need the $||*||_2$ norm or the Frobenius one because I actually need both, so whatever answer with one of them is valid :D )
Thank you!