First of all, let's mention what does *"my neural network doesn't generalize well"* mean and what's the difference with saying *"my neural network doesn't perform well"*.
When training a Neural Network, you are constantly evaluating it on a set of labelled data called the *training set*. If your model isn't working properly and doesn't appear to learn from the training set, you **don't** have a generalization issue yet, instead please refer to [this post](https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn). However, if your model is achieving a satisfactory performance on the training set, but cannot perform well on previously unseen data (e.g. validation/test sets), then you **do** have a generalization problem.
## Why is your model not generalizing properly?
The most important part is understanding **why** your network doesn't generalize well. High-capacity Machine Learning models have the ability to **memorize** the training set, which can lead to **overfitting**.
Overfitting is the state where an estimator has begun to learn the training set so well that it has started to model the **noise** in the training samples (besides all useful relationships).
For example, in the image below we can see how the blue on the right line has clearly overfit.
## But why is this bad?
When attempting to evaluate our model on new, previously **unseen** data (i.e. validation/test set), the model's performance will be **much worse** than what we expect.
## How to prevent overfitting?
In the beginning of the post I implied that the complexity of your model is what is actually causing the overfitting, as it is allowing the model to extract unnecessary relationships from the training set, that map its inherent noise. The easiest way to reduce overfitting is to essentially **limit** the capacity of your model. These techniques are called **regularization** techniques.
- **Parameter norm penalties**. These add an extra term to the weight update function of each model, that is dependent on the norm of the parameters. This term's purpose is to **counter** the actual update (i.e. limit how much each weight can be updated). This makes the models more robust to outliers and noise. Examples of such regularizations are [L1 and L2](http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/) regularizations, which can be found on the [Lasso](https://en.wikipedia.org/wiki/Lasso_(statistics)), [Ridge](https://en.wikipedia.org/wiki/Tikhonov_regularization) and [Elastic Net](https://en.wikipedia.org/wiki/Elastic_net_regularization) regressors.
Since each (fully connected) layer in a neural network functions much like a simple linear regression, these are used in Neural Networks. The most common use is to regularize each layer individually.
[keras implementation](https://keras.io/regularizers/).
- **Early stopping**. This technique attempts to stop an estimator's training phase prematurely, at the point where it has learned to extract all meaningful relationships from the data, before beginning to model its noise. This is done by **monitoring** the **validation loss** (or a validation metric of your choosing) and **terminating** the training phase when this metric **stops improving**. This way we give the estimator enough time to learn the useful information but not enough to learn from the noise.
[keras implementation](https://keras.io/callbacks/#earlystopping).
- **Neural Network specific** regularizations. Some examples are:
- **Dropout**. [Dropout](http://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf?utm_content=buffer79b43&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer) is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are **dropped**. This causes the subsequent layer rely on **all** of its connections to the previous layer.
[keras implementation](https://keras.io/layers/core/#dropout)
- **Transfer learning**. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture **pre-trained** on a large, generic dataset.
- Other things that may limit overfitting in Deep Neural Networks are: **[Batch Normalization](https://en.wikipedia.org/wiki/Batch_normalization)**, which can act as a regulizer and in some cases (e.g. inception modules) works as well as dropout; relatively **[small sized batches](https://arxiv.org/pdf/1705.08741.pdf)** in SGD, which can also prevent overfitting; [adding small random noise to weights](https://pdfs.semanticscholar.org/1023/043d2a5a76a07388c3e17c1284018937dbfc.pdf) in hidden layers.
Another way of preventing overfitting, besides limiting the model's capacity, is by improving the quality of your data. The most obvious choice would be **outlier/noise** removal, however in practice their usefulness is limited. A more common way (especially in image-related tasks) is **data augmentation**. Here we attempt randomly transform the training examples so that while they appear to the model to be different, they convey the same semantic information (e.g. left-right flipping on images).
[Data augmentation overview](http://cs231n.stanford.edu/reports/2017/pdfs/300.pdf)
### Practical suggestions:
- By far the most effective regularization technique is **dropout**, meaning that it should be the first you should use. However, you don't need to (and probably shouldn't) place dropout everywhere! The most prone layers to overfitting are the Fully Connected (FC) layers, because they contain the most parameters. Dropout should be applied to these layers (impacting their connections to the next layer).
- **Batch normalization**, besides having a regularization effect aids your model in several other ways (e.g. speeds up convergence, allows for the use of higher learning rates). It too should be used in FC layers.
- As mentioned previously it also may be beneficial to stop your model earlier in the training phase than scheduled. The problem with **early stopping** is that there is no guarantee that, at any given point, the model won't start improving again. A more practical approach than early stopping is storing the weights of the model that achieve the best performance on the validation set. Be cautious, however, as this is not an unbiased estimate of the performance of your model (just better than the training set). You **can** also overfit on the validation set. More on that later.
[keras implementation](https://keras.io/callbacks/#modelcheckpoint)
- In some applications (e.g. image related tasks), it is highly recommended to follow an already established architecture (e.g. VGG, ResNet, Inception), that you can find ImageNet weights for. The generic nature of this dataset, allows the features to be in turn generic enough to be used for any image related task. Besides being robust to overfitting this will greatly reduce the training time.
Another use of the similar concept is the following: if your task doesn't have much data, but you can find another similar task that does, you can use **transfer learning** to reduce overfitting. First train your network for the task that has the larger dataset and then attempt to **fine-tune** the model to the one you initially wanted. The initial training will, in most cases, make your model more robust to overfitting.
- **Data augmentation**. While it **always** helps to have a larger dataset, data augmentation techniques do have their shortcomings. More specifically, you have to be careful not to augment *too strongly*, as this might ruin the semantic content of the data. For example in image augmentation if you translate/shift/scale or adjust the brighness/contrast the image too much you'll lose much of the information it contains. Furthermore, augmentation schemes need to be implemented for each task in an ad-hoc fashion (e.g. in handwritten digit recognition the digits are usually aligned and shouldn't be rotated too much; also they shouldn't be flipped in any direction, as they aren't horizontally/vertically symetric. Same goes for medical images).
In short be careful not to produce **non realistic** images through data augmentation. Moreover, an increased dataset size will require a longer training time. Personally, I start considering using data augmentation when I see that my model is reaching near $0$ loss on the training set.