# What is the difference between kernel, bias, and activity regulizers, and when to use which?

I've read this post, but I wanted more clarification for a broader question.

In Keras, there are now three types of regularizers for a layer: kernel_regularizer, bias_regularizer, activity_regularizer.

I have read posts that explain the difference between L1 and L2 norm, but in an intuitive sense, I'd like to know how each regularizer will affect the aforementioned three types of regularizers and when to use what.

The motivation for my question is that my understanding is that regularizers are usually applied to the loss function. However, they're even being added to bias term. I'm not able to wrap my head around why one would think to do this, let alone be able to discern when to use L1 and L2 for the bias regularizer. Hence, I wanted to get an overall understanding of all three entities that regularizers are applied on and in general know how the 2 kinds of regularizers can affect each of those entities at a high level.

### What is the difference between them?

You have the regression equation $$y = Wx+b$$, where $$x$$ is the input, $$W$$ the weights matrix and $$b$$ the bias.

• Kernel Regularizer: Tries to reduce the weights $$W$$ (excluding bias).
• Bias Regularizer: Tries to reduce the bias $$b$$.
• Activity Regularizer: Tries to reduce the layer's output $$y$$, thus will reduce the weights and adjust bias so $$Wx+b$$ is smallest.

### When to use which?

Usually, if you have no prior on the distribution that you wish to model, you would only use the kernel regularizer, since a large enough network can still model your function even if the regularization on the weights are big.

If you want the output function to pass through (or have an intercept closer to) the origin, you can use the bias regularizer.

If you want the output to be smaller (or closer to 0), you can use the activity regularizer.

### $$L_1$$ versus $$L_2$$ regularization

Now, for the $$L_1$$ versus $$L_2$$ loss for weight decay (not to be confused with the outputs loss function).

• $$L_2$$ loss is defined as $$w^2$$
• $$L_1$$ loss is defined as $$|w|$$.

where $$w$$ is a component of the matrix $$W$$.

• The gradient of $$L_2$$ will be: $$2w$$
• The gradient of $$L_1$$ will be: $$sign(w)$$

Thus, for each gradient update with a learning rate $$a$$, in $$L_2$$ loss, the weights will be subtracted by $$aW$$, while in $$L_1$$ loss they will be subtracted by $$a \cdot sign(W)$$.

The effect of $$L_2$$ loss on the weights is a reduction of large components in the matrix $$W$$, while $$L_1$$ loss will make the weights matrix sparse, with many zero values. The same applies to the bias and output respectively using the bias and activity regularizer.

• Can you expand on the last paragraph you wrote? Particularly, what is the consequence of the network when there is reduction in the large components (does the network become more erratic to small changes since the big components no longer over shadow the smaller ones) and what does it mean if there are more 0 values? Does that mean the network becomes less erratic? Dec 20 '18 at 2:20
• The network will be less sensitive to small changes (eg. noise) since the matrix $W$ will have small weights. Dec 20 '18 at 17:20
• Sorry, I'm slightly confused. You previous comment refers to which loss, L1? Dec 20 '18 at 17:35
• If you apply $L2$ weight decay, the network will try to be less sensitive to small changes. If you apply $L1$ weight decay, it goes further, the network will try to ignore some inputs altogether. Dec 20 '18 at 19:05
• Please look at this demonstration, maybe it will be clearer. nghiaho.com/?p=1796 Dec 20 '18 at 19:06

kernel_regularizer acts on the weights, while bias_initializer acts on the bias and activity_regularizer acts on the y(layer output).

We apply kernel_regularizer to penalize the weights which are very large causing the network to overfit, after applying kernel_regularizer the weights will become smaller.

While we bias_regularizer to add a bias so that our bias approaches towards zero.

activity_regularizer tries to make the output smaller so as to remove overfitting.

I will expand upon @Bloc97 's answer about the difference between $$L1$$ and $$L2$$ constraints, in order to show why $$L1$$ may drive some weights to zero.

In the case of $$L2$$ regularization, the gradient of a single weight is given by $$\delta w = u - 2pw$$ where $$u$$ is the input from the previous layer being multiplied by weight $$w$$, and $$p$$ is parameter weighting the $$L2$$ penalty.

Without loss of generalization, assume that $$u>0$$ and $$w>0$$.

Then the sign of $$\delta w$$ is given by $$sign(\delta w) = sign(\frac{u}{2p} -w)$$ showing that $$L2$$ regularization will drive $$w$$ to grow bigger if $$w$$ drops below $$\frac{u}{2p}$$.

On the other hand, in the case of $$L1$$ regularization, the gradient of a single weight is given by $$\delta w = u - p$$ so the sign of $$\delta w$$ is given by $$sign(\delta w) = sign(u-p)$$ showing that $$L1$$ regularization will drive $$w$$ to grow smaller when the input $$u$$ is smaller than the $$L1$$ regularization parameter $$p$$.

Effectively, $$p$$ is functioning as a threshold such that, whenever $$u$$ is less than $$p$$, $$L1$$ regularization will push the weight to grow smaller, and whenever $$u$$ is greater than $$p$$, $$L1$$ regularization will push the weight to grow larger.

The above is a local linear approximation of a nonlinear system: $$u$$ is actually an average over, for example, all the samples in a batch, and $$u$$ also changes with with each update. Nevertheless, it gives an intuitive understanding of how $$L1$$ regularization tries to drive some weights to zero (given large enough $$p$$).