Return to Answer

For example, in the image below we can see how the blue line has clearly overfit.

For example, in the image below we can see how the blue on the right line has clearly overfit.

• Parameter norm penalties. These add an extra term to the weight update function of each model, that is dependent on the norm of the parameters. This term's purpose is to counter the actual update (i.e. limit how much each weight can be updated). This makes the models more robust to outliers and noise. Examples of such regularizations are L1 and L2 regularizations, which can be found on the Lasso, Ridge and Elastic Net regressors.

• Early stopping. This technique attempts to stop an estimator's training phase prematurely, at the point where it has learned to extract all meaningful relationships from the data, before beginning to model its noise. This is done by monitoring the validation loss (or a validation metric of your choosing) and terminating the training phase when this metric stops improving. This way we give the estimator enough time to learn the useful information but not enough to learn from the noise.

• Neural Network specific regularizations. Some examples are:
  • Dropout. Dropout is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are dropped. This causes the subsequent layer rely on all of its connections to the previous layer.
  • Transfer learning. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture pre-trained on a large, generic dataset.
  • Other things that may limit overfitting in Deep Neural Networks are: Batch Normalization, which can act as a regulizer; relatively small sized batches in SGD, which can also prevent overfitting; adding small random noise to hidden layers.

Another way of preventing overfitting, besides limiting the model's capacity, is by improving the quality of your data. The most obvious choice would be outlier/noise removal, however in practice their usefulness is limited. A more common way (especially in image-related tasks) is data augmentation. Here we attempt randomly transform the training examples so that while they appear to the model to be different, they convey the same semantic information (e.g. left-right flipping on inages).

First of all, let's mention what does "my neural network doesn't generalize well" mean and what's the difference with saying "my neural network doesn't perform well".

When training a Neural Network, you are constantly evaluating it on a set of labelled data called the training set. If your model isn't working properly and doesn't appear to learn from the training set, you don't have a generalization issue yet, instead please refer to this post. However, if your model is achieving a satisfactory performance on the training set, but cannot perform well on previously unseen data (e.g. validation/test sets), then you do have a generalization problem.

• Parameter norm penalties. These add an extra term to the weight update function of each model, that is dependent on the norm of the parameters. This term's purpose is to counter the actual update (i.e. limit how much each weight can be updated). This makes the models more robust to outliers and noise. Examples of such regularizations are L1 and L2 regularizations, which can be found on the Lasso, Ridge and Elastic Net regressors.
  Since each (fully connected) layer in a neural network functions much like a simple linear regression, these are used in Neural Networks. The most common use is to regularize each layer individually.
  keras implementation.

• Early stopping. This technique attempts to stop an estimator's training phase prematurely, at the point where it has learned to extract all meaningful relationships from the data, before beginning to model its noise. This is done by monitoring the validation loss (or a validation metric of your choosing) and terminating the training phase when this metric stops improving. This way we give the estimator enough time to learn the useful information but not enough to learn from the noise.
  keras implementation.

• Neural Network specific regularizations. Some examples are:
  • Dropout. Dropout is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are dropped. This causes the subsequent layer rely on all of its connections to the previous layer.
    keras implementation
  • Transfer learning. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture pre-trained on a large, generic dataset.
  • Other things that may limit overfitting in Deep Neural Networks are: Batch Normalization, which can act as a regulizer and in some cases (e.g. inception modules) works as well as dropout; relatively small sized batches in SGD, which can also prevent overfitting; adding small random noise to weights in hidden layers.

Another way of preventing overfitting, besides limiting the model's capacity, is by improving the quality of your data. The most obvious choice would be outlier/noise removal, however in practice their usefulness is limited. A more common way (especially in image-related tasks) is data augmentation. Here we attempt randomly transform the training examples so that while they appear to the model to be different, they convey the same semantic information (e.g. left-right flipping on images).
Data augmentation overview

Practical suggestions:

• By far the most effective regularization technique is dropout, meaning that it should be the first you should use. However, you don't need to (and probably shouldn't) place dropout everywhere! The most prone layers to overfitting are the Fully Connected (FC) layers, because they contain the most parameters. Dropout should be applied to these layers (impacting their connections to the next layer).
• Batch normalization, besides having a regularization effect aids your model in several other ways (e.g. speeds up convergence, allows for the use of higher learning rates). It too should be used in FC layers.
• As mentioned previously it also may be beneficial to stop your model earlier in the training phase than scheduled. The problem with early stopping is that there is no guarantee that, at any given point, the model won't start improving again. A more practical approach than early stopping is storing the weights of the model that achieve the best performance on the validation set. Be cautious, however, as this is not an unbiased estimate of the performance of your model (just better than the training set). You can also overfit on the validation set. More on that later.
  keras implementation
• In some applications (e.g. image related tasks), it is highly recommended to follow an already established architecture (e.g. VGG, ResNet, Inception), that you can find ImageNet weights for. The generic nature of this dataset, allows the features to be in turn generic enough to be used for any image related task. Besides being robust to overfitting this will greatly reduce the training time.
  Another use of the similar concept is the following: if your task doesn't have much data, but you can find another similar task that does, you can use transfer learning to reduce overfitting. First train your network for the task that has the larger dataset and then attempt to fine-tune the model to the one you initially wanted. The initial training will, in most cases, make your model more robust to overfitting.
• Data augmentation. While it always helps to have a larger dataset, data augmentation techniques do have their shortcomings. More specifically, you have to be careful not to augment too strongly, as this might ruin the semantic content of the data. For example in image augmentation if you translate/shift/scale or adjust the brighness/contrast the image too much you'll lose much of the information it contains. Furthermore, augmentation schemes need to be implemented for each task in an ad-hoc fashion (e.g. in handwritten digit recognition the digits are usually aligned and shouldn't be rotated too much; also they shouldn't be flipped in any direction, as they aren't horizontally/vertically symetric. Same goes for medical images).
  In short be careful not to produce non realistic images through data augmentation. Moreover, an increased dataset size will require a longer training time. Personally, I start considering using data augmentation when I see that my model is reaching near $0$ loss on the training set.

• Model specific regularizations. Some examples are:
  • Pruning and maximum depth constraints for tree-based algorithms.
  • Dropout for Neural Networks. Dropout is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are dropped. This causes the subsequent layer rely on all of its connections to the previous layer.
  • Transfer learning. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture pre-trained on a large, generic dataset.
  • Other things that may limit overfitting in Deep Neural Networks are: Batch Normalization, which can act as a regulizer; relatively small sized batches in SGD, which can also prevent overfitting; adding small random noise to hidden layers.

• Neural Network specific regularizations. Some examples are:
  • Dropout. Dropout is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are dropped. This causes the subsequent layer rely on all of its connections to the previous layer.
  • Transfer learning. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture pre-trained on a large, generic dataset.
  • Other things that may limit overfitting in Deep Neural Networks are: Batch Normalization, which can act as a regulizer; relatively small sized batches in SGD, which can also prevent overfitting; adding small random noise to hidden layers.