11
$\begingroup$

Here is a keras code sample that uses it:

from keras.constraints import max_norm

model.add(Convolution2D(32, 3, 3, input_shape=(3, 32, 32), 
                        border_mode='same', activation='relu', kernel_constraint=max_norm(3)))
$\endgroup$
15
$\begingroup$

From http://cs231n.github.io/neural-networks-2/#reg:

Max norm constraints. Another form of regularization is to enforce an absolute upper bound on the magnitude of the weight vector for every neuron and use projected gradient descent to enforce the constraint. In practice, this corresponds to performing the parameter update as normal, and then enforcing the constraint by clamping the weight vector $\vec{w}$ of every neuron to satisfy $\Vert \vec{w} \Vert_2 < c.$ Typical values of $c$ are on orders of 3 or 4. Some people report improvements when using this form of regularization. One of its appealing properties is that network cannot “explode” even when the learning rates are set too high because the updates are always bounded.

$\endgroup$
8
$\begingroup$

I found an answer by McLawrence in another question to be very helpful. Reproduced below:

What does a weight constraint of max_normdo?

maxnorm(m) will, if the L2-Norm of your weights exceeds m, scale your whole weight matrix by a factor that reduces the norm to m. As you can find in the keras code in class MaxNorm(Constraint):

def __call__(self, w):
    norms = K.sqrt(K.sum(K.square(w), axis=self.axis, keepdims=True))
    desired = K.clip(norms, 0, self.max_value)
    w *= (desired / (K.epsilon() + norms))
    return w

Aditionally, maxnorm has an axis argument, along which the norm is calculated. In your example you don't specify an axis, thus the norm is calculated over the whole weight matrix. If for example, you want to constrain the norm of every convolutional filter, assuming that you are using tf dimension ordering, the weight matrix will have the shape (rows, cols, input_depth, output_depth). Calculating the norm over axis = [0, 1, 2] will constrain each filter to the given norm.

Why to do it?

Constraining the weight matrix directly is another kind of regularization. If you use a simple L2 regularization term you penalize high weights with your loss function. With this constraint, you regularize directly. As also linked in the keras code, this seems to work especially well in combination with a dropoutlayer. More more info see chapter 5.1 in this paper

$\endgroup$
  • $\begingroup$ Nice answer, but beware: "In your example you don't specify an axis, thus the norm is calculated over the whole weight matrix." - This doens't seem to be true (at least as of now). Rather, the norm is then calculated over the default axis=0. $\endgroup$ – Bobson Dugnutt Nov 24 '18 at 14:37
  • $\begingroup$ I agree, the max-norm is taken as the length of the weights that fan-in to a node in a hidden layer, i.e., $||w||$. Also, did you notice that for dropout regularization, max-norm was used for all runs (see last paragraph on page 1934 of cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf). $\endgroup$ – JoleT Jun 4 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.