There is a method for imposing physical constraints on the neural networks, in which a physics-based loss is added to the loss function. This term is usually a function of the output of the network.
As a simplistic example, assume the network outputs a number, which should not fall below $3$. We add a (penalty) regularization term of the form $\max(3-output, 0)$ that penalizes when output is below $3$ and is canceled out when output is above $3$.
As this is essentially a constrained optimization (minimizing NN loss function subject to the physical constraint above), which is turned into a regularized unconstraint optimization (in the form of a Lagrangian function), we need to find a Lagrangian multiplier for the regularization term.
Question:
I was wondering, in the case of a regularized loss function for a neural network, is it possible to learn the Lagrangian multiplier for the regularization term based on some criteria on the output value while training the network? I have looked at many research papers and it seems everyone is “tuning” the Lagrangian multiplier than learning, which does not guarantee to satisfy KKT conditions.
Ref:
- You can see one such a work here: https://papers.nips.cc/paper/7942-constrained-generation-of-semantically-valid-graphs-via-regularizing-variational-autoencoders.pdf (section 3 and Eq 8 in particular)
