# Regularizing the difference in the norms of two independent weight matrices in a neural network

Say, there are two neural network layers with weights $$W_1$$ and $$W_2$$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could be joined somehow and propagated jointly in subsequent layers.

So, a vanilla L2 term discourages the weight matrices from learning too big values. But if I calculate the Frobenius norm of $$W_1$$ and $$W_2$$, subtract one from the other and add that as a regularization term, what would that mean? Mathematically,

$$L(\centerdot) + \lambda (||W_1||_F - ||W_2||_F)$$ where $$\lambda$$ is some rate parameter.

What would this sort of regularization do to the learning? Can we say this term would encourage $$W_1$$ and $$W_2$$ to be 'similar'? How would it affect the outputs of these layers?

• OP's proposed strategy may result in a model that drives $\|W_1 \|_F$ to zero and/or $\| W_2 \|_F$ to a large value. If they're restricted to be conformable shapes, then you might instead use the penalty $$\lambda \| W_1 - W_2 \|_F;$$ the matrices will be similar in the sense that the differences between them are small.
– Sycorax
May 22, 2020 at 12:59
• Interesting. So if independent inputs undergo a 'similar' non-linear transformation, what kind of relationships (if any) would the outputs have?
– HMK
May 22, 2020 at 13:53
• One way to think about it is that a model which uses $W_1 = W_2$, that is, the same weights, is an extreme of this regularization. My proposal is a relaxation of that. But I don't think you can make general statements about the relationships of the outputs independent of the model, the architecture, and the data.
– Sycorax
May 22, 2020 at 13:58