What is the difference between kernel, bias, and activity regulizers, and when to use which?

Question

I've read this post, but I wanted more clarification for a broader question.

In Keras, there are now three types of regularizers for a layer: kernel_regularizer, bias_regularizer, activity_regularizer.

I have read posts that explain the difference between L1 and L2 norm, but in an intuitive sense, I'd like to know how each regularizer will affect the aforementioned three types of regularizers and when to use what.

The motivation for my question is that my understanding is that regularizers are usually applied to the loss function. However, they're even being added to bias term. I'm not able to wrap my head around why one would think to do this, let alone be able to discern when to use L1 and L2 for the bias regularizer. Hence, I wanted to get an overall understanding of all three entities that regularizers are applied on and in general know how the 2 kinds of regularizers can affect each of those entities at a high level.

Answer 1

What is the difference between them?

You have the regression equation $y = Wx+b$, where $x$ is the input, $W$ the weights matrix and $b$ the bias.

• Kernel Regularizer: Tries to reduce the weights $W$ (excluding bias).
• Bias Regularizer: Tries to reduce the bias $b$.
• Activity Regularizer: Tries to reduce the layer's output $y$, thus will reduce the weights and adjust bias so $Wx+b$ is smallest.

When to use which?

Usually, if you have no prior on the distribution that you wish to model, you would only use the kernel regularizer, since a large enough network can still model your function even if the regularization on the weights are big.

If you want the output function to pass through (or have an intercept closer to) the origin, you can use the bias regularizer.

If you want the output to be smaller (or closer to 0), you can use the activity regularizer.

$L_1$ versus $L_2$ regularization

Now, for the $L_1$ versus $L_2$ loss for weight decay (not to be confused with the outputs loss function).

• $L_2$ loss is defined as $w^2$
• $L_1$ loss is defined as $|w|$.

where $w$ is a component of the matrix $W$.

• The gradient of $L_2$ will be: $2w$
• The gradient of $L_1$ will be: $sign(w)$

Thus, for each gradient update with a learning rate $a$, in $L_2$ loss, the weights will be subtracted by $aW$, while in $L_1$ loss they will be subtracted by $a \cdot sign(W)$.

The effect of $L_2$ loss on the weights is a reduction of large components in the matrix $W$, while $L_1$ loss will make the weights matrix sparse, with many zero values. The same applies to the bias and output respectively using the bias and activity regularizer.

Answer 2

kernel_regularizer acts on the weights, while bias_initializer acts on the bias and activity_regularizer acts on the y(layer output).

We apply kernel_regularizer to penalize the weights which are very large causing the network to overfit, after applying kernel_regularizer the weights will become smaller.

While we bias_regularizer to add a bias so that our bias approaches towards zero.

activity_regularizer tries to make the output smaller so as to remove overfitting.

Answer 3

I will expand upon @Bloc97 's answer about the difference between $L1$ and $L2$ constraints, in order to show why $L1$ may drive some weights to zero.

In the case of $L2$ regularization, the gradient of a single weight is given by $$ \delta w = u - 2pw$$ where $u$ is the input from the previous layer being multiplied by weight $w$, and $p$ is parameter weighting the $L2$ penalty.

Without loss of generalization, assume that $u>0$ and $w>0$.

Then the sign of $\delta w$ is given by $$ sign(\delta w) = sign(\frac{u}{2p} -w)$$ showing that $L2$ regularization will drive $w$ to grow bigger if $w$ drops below $\frac{u}{2p}$.

On the other hand, in the case of $L1$ regularization, the gradient of a single weight is given by $$ \delta w = u - p$$ so the sign of $\delta w$ is given by $$ sign(\delta w) = sign(u-p)$$ showing that $L1$ regularization will drive $w$ to grow smaller when the input $u$ is smaller than the $L1$ regularization parameter $p$.

Effectively, $p$ is functioning as a threshold such that, whenever $u$ is less than $p$, $L1$ regularization will push the weight to grow smaller, and whenever $u$ is greater than $p$, $L1$ regularization will push the weight to grow larger.

The above is a local linear approximation of a nonlinear system: $u$ is actually an average over, for example, all the samples in a batch, and $u$ also changes with with each update. Nevertheless, it gives an intuitive understanding of how $L1$ regularization tries to drive some weights to zero (given large enough $p$).