6
$\begingroup$

Max norm regularization consists in clipping each neuron's weight vector after each training step to ensure that its norm never exceeds some threshold.

I am looking for the original paper that described this technique. Using scholar.google.com, I found a couple of papers that people seem point to in papers about max norm, but they don't seem to correspond to the technique described above. They're about matrix factorization (and way out of my league):

Any idea?

$\endgroup$
6
$\begingroup$

This paper specifically discusses max-norm with SGD. It references Srebro and Shraibman (2005), which makes me think that it/your implementation does in fact correspond to the technique you described.

Max-norm regularization has been previously used in the context of collaborative filtering (Srebro and Shraibman, 2005). It typically improves the performance of stochastic gradient descent training of deep neural nets, even when no dropout is used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.