When discussing linear regression it is well known that you can add regularization terms, such as,

$$\lambda \|w\|^2 \quad \text{(Tikhonov regularization)}$$

to the empirical error/loss function.

However, regularization seems to be under-discussed when it comes to binary/multi-class training.

For example, I've browsed through hundreds of code examples online for CNN training and not one has included a regularization term to the cross-entropy loss function.

This makes me wonder a couple of things:

  1. does adding regularization to the loss functions for binary/multi-class classification training make sense?

  2. if so, what type of regularization makes sense and why?

  3. if not, why not?

Hope someone can answer.

  • 1
    $\begingroup$ Of possible interest: cs231n.github.io/neural-networks-2/#reg $\endgroup$ – Dave Oct 12 '20 at 10:16
  • $\begingroup$ Let us know if you have further questions or need more explanation. If this answer or any other one solved your issue, please mark it as accepted :) $\endgroup$ – Camille Gontier Dec 18 '20 at 10:05

Depending on what you are trying to do with your CNN, regularization may indeed make sense. Pruning your network by regularization to make it sparse has two main advantages:

  • It simplifies the network, making training and computation faster and easier;
  • It prevents overfitting, and allows to make sure your network will generalize well on new data.

An intuitive way to reach these objectives is to perform $L_0$ regularization, which penalizes parameters than are not strictly equal to 0. This induces sparsity in the network. This procedure is described in the following paper : https://arxiv.org/abs/1712.01312

The authors also discuss other kinds of regularization (namely $L_1$ regularization).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.