What should I do when my neural network doesn't generalize well?

Question

I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar architectures and data. How can I fix this?

---

As for question

What should I do when my neural network doesn't learn?

to which this question is inspired, the question is intentionally left general so that other questions about how to reduce the generalization error of a neural network down to a level which has been proved to be attainable, can be closed as duplicates of this one.

See also dedicated thread on Meta:

Is there a generic question to which we can redirect questions of the type "why does my neural network not generalize well?"

Answer 1

First of all, let's mention what does "my neural network doesn't generalize well" mean and what's the difference with saying "my neural network doesn't perform well".

When training a Neural Network, you are constantly evaluating it on a set of labelled data called the training set. If your model isn't working properly and doesn't appear to learn from the training set, you don't have a generalization issue yet, instead please refer to this post. However, if your model is achieving a satisfactory performance on the training set, but cannot perform well on previously unseen data (e.g. validation/test sets), then you do have a generalization problem.

Why is your model not generalizing properly?

The most important part is understanding why your network doesn't generalize well. High-capacity Machine Learning models have the ability to memorize the training set, which can lead to overfitting.

Overfitting is the state where an estimator has begun to learn the training set so well that it has started to model the noise in the training samples (besides all useful relationships).

For example, in the image below we can see how the blue on the right line has clearly overfit.

But why is this bad?

When attempting to evaluate our model on new, previously unseen data (i.e. validation/test set), the model's performance will be much worse than what we expect.

How to prevent overfitting?

In the beginning of the post I implied that the complexity of your model is what is actually causing the overfitting, as it is allowing the model to extract unnecessary relationships from the training set, that map its inherent noise. The easiest way to reduce overfitting is to essentially limit the capacity of your model. These techniques are called regularization techniques.

• Parameter norm penalties. These add an extra term to the weight update function of each model, that is dependent on the norm of the parameters. This term's purpose is to counter the actual update (i.e. limit how much each weight can be updated). This makes the models more robust to outliers and noise. Examples of such regularizations are L1 and L2 regularizations, which can be found on the Lasso, Ridge and Elastic Net regressors.
  Since each (fully connected) layer in a neural network functions much like a simple linear regression, these are used in Neural Networks. The most common use is to regularize each layer individually.
  keras implementation.

• Early stopping. This technique attempts to stop an estimator's training phase prematurely, at the point where it has learned to extract all meaningful relationships from the data, before beginning to model its noise. This is done by monitoring the validation loss (or a validation metric of your choosing) and terminating the training phase when this metric stops improving. This way we give the estimator enough time to learn the useful information but not enough to learn from the noise.
  keras implementation.

• Neural Network specific regularizations. Some examples are:
  • Dropout. Dropout is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are dropped. This causes the subsequent layer rely on all of its connections to the previous layer.
    keras implementation
  • Transfer learning. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture pre-trained on a large, generic dataset.
  • Other things that may limit overfitting in Deep Neural Networks are: Batch Normalization, which can act as a regulizer and in some cases (e.g. inception modules) works as well as dropout; relatively small sized batches in SGD, which can also prevent overfitting; adding small random noise to weights in hidden layers.

Another way of preventing overfitting, besides limiting the model's capacity, is by improving the quality of your data. The most obvious choice would be outlier/noise removal, however in practice their usefulness is limited. A more common way (especially in image-related tasks) is data augmentation. Here we attempt randomly transform the training examples so that while they appear to the model to be different, they convey the same semantic information (e.g. left-right flipping on images).
Data augmentation overview

Practical suggestions:

• By far the most effective regularization technique is dropout, meaning that it should be the first you should use. However, you don't need to (and probably shouldn't) place dropout everywhere! The most prone layers to overfitting are the Fully Connected (FC) layers, because they contain the most parameters. Dropout should be applied to these layers (impacting their connections to the next layer).
• Batch normalization, besides having a regularization effect aids your model in several other ways (e.g. speeds up convergence, allows for the use of higher learning rates). It too should be used in FC layers.
• As mentioned previously it also may be beneficial to stop your model earlier in the training phase than scheduled. The problem with early stopping is that there is no guarantee that, at any given point, the model won't start improving again. A more practical approach than early stopping is storing the weights of the model that achieve the best performance on the validation set. Be cautious, however, as this is not an unbiased estimate of the performance of your model (just better than the training set). You can also overfit on the validation set. More on that later.
  keras implementation
• In some applications (e.g. image related tasks), it is highly recommended to follow an already established architecture (e.g. VGG, ResNet, Inception), that you can find ImageNet weights for. The generic nature of this dataset, allows the features to be in turn generic enough to be used for any image related task. Besides being robust to overfitting this will greatly reduce the training time.
  Another use of the similar concept is the following: if your task doesn't have much data, but you can find another similar task that does, you can use transfer learning to reduce overfitting. First train your network for the task that has the larger dataset and then attempt to fine-tune the model to the one you initially wanted. The initial training will, in most cases, make your model more robust to overfitting.
• Data augmentation. While it always helps to have a larger dataset, data augmentation techniques do have their shortcomings. More specifically, you have to be careful not to augment too strongly, as this might ruin the semantic content of the data. For example in image augmentation if you translate/shift/scale or adjust the brighness/contrast the image too much you'll lose much of the information it contains. Furthermore, augmentation schemes need to be implemented for each task in an ad-hoc fashion (e.g. in handwritten digit recognition the digits are usually aligned and shouldn't be rotated too much; also they shouldn't be flipped in any direction, as they aren't horizontally/vertically symetric. Same goes for medical images).
  In short be careful not to produce non realistic images through data augmentation. Moreover, an increased dataset size will require a longer training time. Personally, I start considering using data augmentation when I see that my model is reaching near $0$ loss on the training set.

Answer 2

There is plenty of empirical evidence that deep enough neural networks can memorize random labels on huge datasets (Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals, "Understanding deep learning requires rethinking generalization"). Thus in principle by getting a big enough NN we can always reduce the training error to extremely small values, limited in practice by numerical accuracy, no matter how meaningless the task.

Things are quite different for the generalization error. We cannot be sure that for each learning problem, there exists a learnable NN model which can produce a generalization error as low as desired. For this reason the first step is to

1. Set your expectations correctly

Find a reputable reference which tells you that there exists an architecture which can reach the generalization error you're looking for, on your data set or on the most similar one for which you can find references. For example, look here

What are the current state-of-the-art convolutional neural networks?

to find current (at the time of the answers) SOTA (State Of The Art) performance for CNNs on various tasks. It's a good idea to try to reproduce such results on these reference data sets, before you train on your own data set, as a test that all your infrastructure is properly in place.

2. Make sure your training procedure is flawless

All the checks described in the answers to question

What should I do when my neural network doesn't learn?

to make sure that your training procedure is ok, are a prerequisite for successful reduction of the generalisation error (if your NN is not learning, it cannot learn to generalise). These checks include, among the other stuff:

• unit tests
• dataset checks (have a look at a few random input/label samples for both the training set and test set and check that the labels are correct; check width and size of input images; shuffle samples in training/test set and see if it affects results; etc.)
• randomisation tests
• standardize your preprocessing and package versions
• keep a logbook of numerical experiments

3. Try to get superconvergence

“Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates” by Leslie N. Smith and Nicholay Topin shows that in some cases the combination of large learning rates with the cyclical learning rate method of Leslie N. Smith acts as a regulariser, accelerating convergence by an order of magnitude and reducing the need for extensive regularisation. Thus this is a good thing to try before

4. Setting your regularisation to the MAXXX

Regularisation often increases training time (bad), increases the training error and reduces the generalisation error (good), but too much regularisation can actually increase both errors (underfitting). For this reason, and because of the increase in training time, it’s often better to introduce the various regularisation techniques one at a time, after you successfully managed to overfit the training set. Note that regularisation by itself doesn’t necessarily imply your generalisation error will get smaller: the model must have a large enough capacity to achieve good generalisation properties. This often means that you need a sufficiently deep network, before you can see the benefits of regularisation.

The oldest regularisation methods are probably early stopping and weight decay. Some of the others:

• reduce batch size: smaller batch sizes are usually associated with smaller generalisation error, so this is something to try. However, note that some dispute the usefulness of minibatches: in my experience, they help (as long as you don’t have to use crazy small sizes such as $m=16$), but Elad Hoffer, Itay Hubara, Daniel Soudry Train longer, generalize better: closing the generalization gap in large batch training of neural networks disagree. Note that if you use batch norm (see below), too small minibatches will be quite harmful.
• use SGD rather than adaptive optimisers: this has been already covered by @shimao, thus I only mention it for the sake of completeness
• use dropout: if you use LSTMs, use standard dropout only for input and output units of a LSTM layer. For the recurrent units (the gates) use recurrent dropout, as first shown by Yarin Gal in his Ph.D. thesis. However, if you use CNNs, dropout is used less frequently now. Instead, you tend to…
• ...use batch normalisation: the most recent CNN architectures eschew dropout in favour of batch normalisation. This could be just a fad, or it could be due to the fact that apparently dropout and batch normalisation don’t play nice together (Xiang Li, Shuo Chen, Xiaolin Hu, Jian Yang, Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift). Since batch norm is more effective than dropout when you have huge data sets, this could be a reason why dropout has fallen out of favour for CNN architectures. If you use batch normalisation, verify that the distribution of weights and biases for each layer looks approximately standard normal. For RNNs, implementing batch norm is complicated: weight normalisation (Tim Salimans, Diederik P. Kingma, Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks) is a viable alternative.
• use data augmentation: it also has a regularising effect.

5. Hyperparameter/architecture search

If nothing else helps, you will have to test multiple different hyperparameter settings (Bayesian Optimization may help here) or multiple different architectural changes (e.g. maybe in your GAN architecture and for the data set you're working on, batch norm only works in the generator, but when added to the discriminator too it makes things worse). Be sure to keep track of the results of these long and boring experiments in a well-ordered logbook.

PS for a GAN it doesn't make much sense to talk about a generalization error: the above example was meant only as an indication that there's still a lot of alchemy in Deep Learning, and things that you would expect to work fine, sometimes don't, or vice versa something which worked ok many times, suddenly craps out on you for a new data set.

Answer 3

A list of commonly used regularization techniques which I've seen in the literature are:

1. Using batch normalization, which is a surprisingly effective regularizer to the point where I rarely see dropout used anymore, because it is simply not necessary.
2. A small amount of weight decay.
3. Some more recent regularization techniques include Shake-shake ("Shake-Shake regularization" by Xavier Gastaldi) and Cutout ( "Improved Regularization of Convolutional Neural Networks with Cutout" by Terrance DeVries and Graham W. Taylor). In particular, the ease with which Cutout can be implemented makes it very attractive. I believe these work better than dropout -- but I'm not sure.
4. If possible, prefer fully convolutional architectures to architectures with fully connected layers. Compare VGG-16, which has 100 million parameters in a single fully connected layer, to Resnet-152, which has 10 times the number of layers and still fewer parameters.
5. Prefer SGD to other optimizers such as Rmsprop and Adam. It has been shown to generalize better. ("Improving Generalization Performance by Switching from Adam to SGD" by Nitish Shirish Keskar and Richard Socher)

Answer 4

I feel like Djib2011, give great points about automated methods, but they don't really tackle the underlying issue of how do we know if the method employed to reduce overfitting did its job. So as an important footnote to DeltaIV answer, I wanted to include this based on recent research in the last 2 years. Overfitting for neural networks isn't just about the model over-memorizing, its also about the models inability to learn new things or deal with anomalies.

Detecting Overfitting in Black Box Model: Interpretability of a model is directly tied to how well you can tell a models ability to generalize. Thus many interpretable plots are methods of detecting overfitting and can tell you how well any of the methods suggested above are working. Interpretability plots directly detect it especially if you compare the validation and test result plots. Chapters 5 and 6 of this unpublished book talk about recent advances in the field detection of overfitting: Interpretable Modeling

Based on this book, I would like to mention three other methods of detecting and removing overfitting, that might be obvious to some, but I personally find that people forget these too often. So I would like to emphasize them if not one minds:

1. Feature Selection Detection: The less number of parameters and less features your model has the better. So if you only include the important one's of the 100 million (maybe have 75 million instead), you will have a better generalizable model. The problem is many neural networks are not perfect in feature selection especially when # 2 is present. Bootstrap or Boosting fundamentally cannot fix both (only a version called wild bootstrap can). In simpler terms, If you give you neural network junk data then it's going to give you junk out. (L2 Normalization mentioned above is very good at helping with this)

2. Detection and Dealing with Anomalies: The fewer "outliers" the more generalizable the model. By "outliers", we don't mean just outliers in the data. Outliers in the data (like the kind you see with a box plot) is a too narrow definition for neural networks. You need to consider also outliers in the error in a model, which is referred to as influence, as well as other anomalies. So detecting anomalies before you run your network is important. A neural net can be robust against one type of anomaly, but robust not against all other types. Counter Example methods, Criticism methods, and Adversarial example methods, and Influence plots are great at helping you discover outliers, and then figure out how to factor them in. (Ie. change the parameters or even remove some of the data)

3. Stratified Sampling, Oversampling, and Undersampling based on statistical or ethical considerations: I wish i was an expert in under and oversampling, but I am not but I know about stratified sampling. Clustering important factors such as (race, sex, gender) and then doing stratified sampling by the cluster is vital to not overfit when one considers big data. When doing image detection, stratified sampling in combination with clustering is legally required in some fields to avoid racial discrimination. The book linked above briefly talks about a methods to do this.

P.S. Should I include more links?

Answer 5

Reduce the number of parameters in the model.

The existing answers focus on different regularization strategies that can improve fit, given a model architecture that remains fixed (same configuration, number of layers, number of neurons in each layer).

However, the simplest and easiest step to reducing overfitting in a neural network is to reduce the number of parameters in the model. This can mean some combination of

• fewer layers in the model; and
• fewer parameters in each layer.

This can reduce overfitting because a model with a larger parameter count has a greater flexibility to fit the data. Intuitively, this is analogous to the simpler case of adding degrees of freedom to a linear model. A linear model with at least as many degrees of freedom as the number of observations can achieve a perfect fit to the training data, because it can interpolate between each training data point. However, this is unlikely to generalize well because, by perfectly interpolating the training data, the model has also fit to the noise in the target variable.

From an overfitting perspective, the goal of adjusting the number of parameters in the model is to achieve the correct trade-off between achieving a good fit to the data and a fit that will generalize to new data.