Say, there are two neural network layers with weights $W_1$ and $W_2$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could be joined somehow and propagated jointly in subsequent layers.

So, a vanilla L2 term discourages the weight matrices from learning too big values. But if I calculate the Frobenius norm of $W_1$ and $W_2$, subtract one from the other and add that as a regularization term, what would that mean? Mathematically,

$L(\centerdot) + \lambda (||W_1||_F - ||W_2||_F) $ where $\lambda$ is some rate parameter.

What would this sort of regularization do to the learning? Can we say this term would encourage $W_1$ and $W_2$ to be 'similar'? How would it affect the outputs of these layers?

  • $\begingroup$ OP's proposed strategy may result in a model that drives $\|W_1 \|_F$ to zero and/or $\| W_2 \|_F$ to a large value. If they're restricted to be conformable shapes, then you might instead use the penalty $$\lambda \| W_1 - W_2 \|_F;$$ the matrices will be similar in the sense that the differences between them are small. $\endgroup$
    – Sycorax
    May 22, 2020 at 12:59
  • $\begingroup$ Interesting. So if independent inputs undergo a 'similar' non-linear transformation, what kind of relationships (if any) would the outputs have? $\endgroup$
    – HMK
    May 22, 2020 at 13:53
  • $\begingroup$ One way to think about it is that a model which uses $W_1 = W_2$, that is, the same weights, is an extreme of this regularization. My proposal is a relaxation of that. But I don't think you can make general statements about the relationships of the outputs independent of the model, the architecture, and the data. $\endgroup$
    – Sycorax
    May 22, 2020 at 13:58


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.