6
$\begingroup$

I've read this post, but I wanted more clarification for a broader question.

In Keras, there are now three types of regularizers for a layer: kernel_regularizer, bias_regularizer, activity_regularizer.

I have read posts that explain the difference between L1 and L2 norm, but in an intuitive sense, I'd like to know how each regularizer will affect the aforementioned three types of regularizers and when to use what.

The motivation for my question is that my understanding is that regularizers are usually applied to the loss function. However, they're even being added to bias term. I'm not able to wrap my head around why one would think to do this, let alone be able to discern when to use L1 and L2 for the bias regularizer. Hence, I wanted to get an overall understanding of all three entities that regularizers are applied on and in general know how the 2 kinds of regularizers can affect each of those entities at a high level.

$\endgroup$
9
$\begingroup$

You have the regression equation $y = Wx+b$, where $x$ is the input, $W$ the weights matrix and $b$ the bias.

  • Kernel Regularizer: Tries to reduce the weights $W$ (excluding bias).
  • Bias Regularizer: Tries to reduce the bias $b$.
  • Activity Regularizer: Tries to reduce the layer's output $y$, thus will reduce the weights and adjust bias so $Wx+b$ is smallest.

Usually if you have no prior on the distribution that you wish to model, you would only use the Kernel Regularizer, since a large enough network can still model your function even if the regularization on the weights are big.

If you want the output function to pass through (or have an intercept closer to) the origin, you can use the Bias Regularizer.
If you want the output to be smaller (or closer to 0), you can use the Activity Regularizer.

Now for the $L1$ versus $L2$ loss for weight decay (not to be confused with the outputs loss function).
$L2$ loss is defined as $w^2$
$L1$ loss is defined as $|w|$.
$w$ is a component of the matrix $W$.

The gradient of $L2$ will be: $2w$
The gradient of $L1$ will be: $sign(w)$

Thus, for each gradient update with a learning rate $a$, in $L2$ loss, the weights will be subtracted by $aW$, while in $L1$ loss they will be subtracted by $a \cdot sign(W)$.

The effect of $L2$ loss on the weights is a reduction of large components in the matrix $W$ while $L1$ loss will make the weights matrix sparse, with many zero values. The same applies on the bias and output respectively using the bias and activity regularizer.

$\endgroup$
  • $\begingroup$ Can you expand on the last paragraph you wrote? Particularly, what is the consequence of the network when there is reduction in the large components (does the network become more erratic to small changes since the big components no longer over shadow the smaller ones) and what does it mean if there are more 0 values? Does that mean the network becomes less erratic? $\endgroup$ – Christian Dec 20 '18 at 2:20
  • $\begingroup$ The network will be less sensitive to small changes (eg. noise) since the matrix $W$ will have small weights. $\endgroup$ – Bloc97 Dec 20 '18 at 17:20
  • $\begingroup$ Sorry, I'm slightly confused. You previous comment refers to which loss, L1? $\endgroup$ – Christian Dec 20 '18 at 17:35
  • $\begingroup$ If you apply $L2$ weight decay, the network will try to be less sensitive to small changes. If you apply $L1$ weight decay, it goes further, the network will try to ignore some inputs altogether. $\endgroup$ – Bloc97 Dec 20 '18 at 19:05
  • $\begingroup$ Please look at this demonstration, maybe it will be clearer. nghiaho.com/?p=1796 $\endgroup$ – Bloc97 Dec 20 '18 at 19:06
0
$\begingroup$

kernel_regularizer acts on the weights, while bias_initializer acts on the bias and activity_regularizer acts on the y(layer output).

We apply kernel_regularizer to punish the weights which are very large causing the network to overfit, after applying kernel_regularizer the weights will become smaller.

While we bias_regularizer to add a bias so that our bias approaches towards zero.

activity_regularizer tries to make the output smaller so as to remove overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.