0
$\begingroup$

Say, there are two neural network layers with weights $W_1$ and $W_2$. These two layers are part of a larger network but their inputs are completely independent of each other and their outputs could be joined somehow and propagated jointly in subsequent layers.

So, a vanilla L2 term discourages the weight matrices from learning too big values. But if I calculate the Frobenius norm of $W_1$ and $W_2$, subtract one from the other and add that as a regularization term, what would that mean? Mathematically,

$L(\centerdot) + \lambda (||W_1||_F - ||W_2||_F) $ where $\lambda$ is some rate parameter.

What would this sort of regularization do to the learning? Can we say this term would encourage $W_1$ and $W_2$ to be 'similar'? How would it affect the outputs of these layers?

Improve this question
$\endgroup$
3
  • $\begingroup$ OP's proposed strategy may result in a model that drives $\|W_1 \|_F$ to zero and/or $\| W_2 \|_F$ to a large value. If they're restricted to be conformable shapes, then you might instead use the penalty $$\lambda \| W_1 - W_2 \|_F;$$ the matrices will be similar in the sense that the differences between them are small. $\endgroup$
    – Sycorax ♦
    May 22, 2020 at 12:59
  • $\begingroup$ Interesting. So if independent inputs undergo a 'similar' non-linear transformation, what kind of relationships (if any) would the outputs have? $\endgroup$
    – HMK
    May 22, 2020 at 13:53
  • $\begingroup$ One way to think about it is that a model which uses $W_1 = W_2$, that is, the same weights, is an extreme of this regularization. My proposal is a relaxation of that. But I don't think you can make general statements about the relationships of the outputs independent of the model, the architecture, and the data. $\endgroup$
    – Sycorax ♦
    May 22, 2020 at 13:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.