# Why not normalizing filters in convolutional networks

I read the answer in How to normalize filters in convolutional neural networks? And I know normalizing kernel is not a convention in neural net training. But what is the reason behind that?

Normalizing batches do exist, and are commonplace in neural networks.

As far as normalizing filters, the way it is defined in the link, it is actually a relaxation of regularization. Note that

$\min_{||x||=1}f(x) \geq min_{||x||\leq1} f(x)$,

where $f()$ i some notion of loss and $x$ is your weight vector

The Lagrangian of the latter formulation is

$f(x) + \lambda ||x-1||$

which is what we see as a varaint of regularization. The hyperparameter "tuning" is basically finding the dual variables by trial and error

• I'm afraid this is the original paper about normalizing the feature maps, i.e, BatchNorm, not about the kernels. Apr 12 '18 at 8:13
• Ahh! Im so sorry..Premature jump to conclusion :). Is this what you are looking for?
– Sid
Apr 12 '18 at 16:29
• Oh, I see. I think the regularization can be regarded as a form of kernel normalization. Apr 13 '18 at 2:17
• BTW, aren't the Lagrangian of these two problem the same? Apr 13 '18 at 2:20
• Not exactly. In the first case, (the equality) the dual variable can be positive or negative and in the latter it has to be strictly positive. The form of the equations will be the same. But, with the inequality constraint the minima is less than or equal to that with the equality constraint
– Sid
Apr 13 '18 at 16:08