In the book Hands-On Machine Learning with Scikit-Learn and TensorFlow, Chapter 1, the author stated that when doing Ridge Regression (for only one predictor), this regularization reduces the risk of overfitting.
We have three models:
- Linear model on partial data
- Regularized linear model on partial data (Ridge regression): we can see that the slope is slightly lower because of the regularization.
- Linear model on all data: then the author add some more data, and say that:
regularization forced the model to have a smaller slope: this model does not fit the training data (circles) as well as the first model, but it actually generalizes better to new examples that it did not see during training (squares).
But these new points (red squares) seem to be anomalies compared to the original data. And if we are talking about anomalies, there can be anomalies in the other way (see green squares).
The code can be seen here