# Does it make sense to regularize the loss function for binary/multi-class classification?

When discussing linear regression it is well known that you can add regularization terms, such as,

$$\lambda \|w\|^2 \quad \text{(Tikhonov regularization)}$$

to the empirical error/loss function.

However, regularization seems to be under-discussed when it comes to binary/multi-class training.

For example, I've browsed through hundreds of code examples online for CNN training and not one has included a regularization term to the cross-entropy loss function.

This makes me wonder a couple of things:

1. does adding regularization to the loss functions for binary/multi-class classification training make sense?

2. if so, what type of regularization makes sense and why?

3. if not, why not?

• Of possible interest: cs231n.github.io/neural-networks-2/#reg – Dave Oct 12 '20 at 10:16
• Let us know if you have further questions or need more explanation. If this answer or any other one solved your issue, please mark it as accepted :) – Camille Gontier Dec 18 '20 at 10:05

An intuitive way to reach these objectives is to perform $$L_0$$ regularization, which penalizes parameters than are not strictly equal to 0. This induces sparsity in the network. This procedure is described in the following paper : https://arxiv.org/abs/1712.01312
The authors also discuss other kinds of regularization (namely $$L_1$$ regularization).