## Why is your model not generalizing properly?
The most important part is understanding **why** your network doesn't generalize well. High-capacity Machine Learning models have the ability to **memorize** the training set, which can lead to **overfitting**.
Overfitting is the state where an estimator has begun to learn the training set so well that it has started to model the **noise** in the training samples (besides all useful relationships).
For example, in the image below we can see how the blue line has clearly overfit.
## But why is this bad?
When attempting to evaluate our model on new, previously **unseen** data (i.e. validation/test set), the model's performance will be **much worse** than what we expect.
## How to prevent overfitting?
In the beginning of the post I implied that the complexity of your model is what is actually causing the overfitting, as it is allowing the model to extract unnecessary relationships from the training set, that map its inherent noise. The easiest way to reduce overfitting is to essentially **limit** the capacity of your model. These techniques are called **regularization** techniques.
- **Parameter norm penalties**. These add an extra term to the weight update function of each model, that is dependent on the norm of the parameters. This term's purpose is to **counter** the actual update (i.e. limit how much each weight can be updated). This makes the models more robust to outliers and noise. Examples of such regularizations are [L1 and L2](http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/) regularizations, which can be found on the [Lasso](https://en.wikipedia.org/wiki/Lasso_(statistics)), [Ridge](https://en.wikipedia.org/wiki/Tikhonov_regularization) and [Elastic Net](https://en.wikipedia.org/wiki/Elastic_net_regularization) regressors.
- **Early stopping**. This technique attempts to stop an estimator's training phase prematurely, at the point where it has learned to extract all meaningful relationships from the data, before beginning to model its noise. This is done by **monitoring** the **validation loss** (or a validation metric of your choosing) and **terminating** the training phase when this metric **stops improving**. This way we give the estimator enough time to learn the useful information but not enough to learn from the noise.
- **Model specific** regularizations. Some examples are:
- **Pruning** and **maximum depth** constraints for **tree-based** algorithms.
- **Dropout** for **Neural Networks**. Dropout is an interesting technique that works surprisingly well. Dropout is applied between two successive layers in a network. At each iteration a specified percentage of the connections (selected randomly), connecting the two layers, are **dropped**. This causes the subsequent layer rely on **all** of its connections to the previous layer.
- **Transfer learning**. This is especially used in Deep Learning. This is done by initializing the weights of your network to the ones of another network with the same architecture **pre-trained** on a large, generic dataset.
- Other things that may limit overfitting in Deep Neural Networks are: **[Batch Normalization](https://en.wikipedia.org/wiki/Batch_normalization)**, which can act as a regulizer; relatively **small sized batches** in SGD, which can also prevent overfitting; adding small random noise to hidden layers.
Another way of preventing overfitting, besides limiting the model's capacity, is by improving the quality of your data. The most obvious choice would be **outlier/noise** removal, however in practice their usefulness is limited. A more common way (especially in image-related tasks) is **data augmentation**. Here we attempt randomly transform the training examples so that while they appear to the model to be different, they convey the same semantic information (e.g. left-right flipping on inages).