I read the answer in How to normalize filters in convolutional neural networks? And I know normalizing kernel is not a convention in neural net training. But what is the reason behind that?


Normalizing batches do exist, and are commonplace in neural networks.

Please see https://arxiv.org/pdf/1502.03167.pdf.

As far as normalizing filters, the way it is defined in the link, it is actually a relaxation of regularization. Note that

$\min_{||x||=1}f(x) \geq min_{||x||\leq1} f(x)$,

where $f()$ i some notion of loss and $x$ is your weight vector

The Lagrangian of the latter formulation is

$f(x) + \lambda ||x-1||$

which is what we see as a varaint of regularization. The hyperparameter "tuning" is basically finding the dual variables by trial and error

  • $\begingroup$ I'm afraid this is the original paper about normalizing the feature maps, i.e, BatchNorm, not about the kernels. $\endgroup$ – Rickyim Apr 12 '18 at 8:13
  • $\begingroup$ Ahh! Im so sorry..Premature jump to conclusion :). Is this what you are looking for? $\endgroup$ – Sid Apr 12 '18 at 16:29
  • $\begingroup$ Oh, I see. I think the regularization can be regarded as a form of kernel normalization. $\endgroup$ – Rickyim Apr 13 '18 at 2:17
  • $\begingroup$ BTW, aren't the Lagrangian of these two problem the same? $\endgroup$ – Rickyim Apr 13 '18 at 2:20
  • $\begingroup$ Not exactly. In the first case, (the equality) the dual variable can be positive or negative and in the latter it has to be strictly positive. The form of the equations will be the same. But, with the inequality constraint the minima is less than or equal to that with the equality constraint $\endgroup$ – Sid Apr 13 '18 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.