Avoid overfitting in regression: alternatives to regularization

Regularization in regression (linear, logistic...) is the most popular way to reduce over-fitting.

When the goal is prediction accuracy (not explaining), are there any good alternatives to regularization, especially suited for big data-sets (mi/billions of observations and millions of features)?

• "Big datasets" may mean a lot of observations, a lot of variables or both, and the answer may depend on the number of observations and variables. – Pere Jul 19 '17 at 10:45
• Why not use norm regularisation? For neural networks , there is dropout – seanv507 Jul 19 '17 at 12:05
• The advantage of regularization is that it's computationally cheap. Ensemble methods such as bagging and boosting (etc.) combined with cross validation methods for model diagnostics are a good alternative, but it will be a much more costly solution. – Digio Jul 19 '17 at 14:04
• This might be of interest: stats.stackexchange.com/a/161592/40604 – Dan Jul 19 '17 at 22:51
• To add to the comment by Digio: regularization is cheap compared to bagging/boosting but still expensive compared to the alternative of "no regularization" (see e.g. this post by Ben Recht on how regularization makes deep learning hard). If you have a huge number of samples, no regularization can work well for far cheaper. The model can still generalize well as @hxd1001 points out) – Berk U. Jul 20 '17 at 19:34

Two important points that are not directly related to your question:

• First, even the goal is accuracy instead of interpretation, regularization is still necessary in many cases, since, it will make sure the "high accuracy" on real testing / production data set, not the data used for modeling.

• Second, if there are billion rows and million columns, it is possible no regularization is needed. This is because the data is huge, and many computational models have "limited power", i.e., it is almost impossible to overfit. This is why some deep neural network has billions of parameters.

Now, about your question. As mentioned by Ben and Andrey, there are some options as alternatives to regularization. I would like to add more examples.

• Use simpler model (For example, reduce number of hidden unit in neural network. Use lower order polynomial kernel in SVM. Reduce number of Gaussians in mixture of Gaussian. etc.)

• Stop early in the optimization. (For example, reduce the epoch in neural network training, reduce number of iterations in optimization (CG, BFGS, etc.)

• Average on many models (For example, random forest etc.)

• Thanks a lot. The second option (stop early) is what we are trying presently with SGD. It works rather well. We want to compare it with regularization soon. Are you aware of any article that mentions this method ? – Benoit Sanchez Jul 19 '17 at 19:47
• There is a hint of a geometric relationship between early stopping with gradient descent, and regularization. For example, ridge regression in it's primal form asks for the parameters minimizing the loss function that lie within a solid ellipse centered at the origin, with the size of the ellipse a function of the regularization strength. The ridge parameters lie on the surface of the ellipse if they are different than the un-regularized solution. If you run an ascent starting at the origin, and then stop early, you will be on the boundary of one of these ellipses... – Matthew Drury Jul 19 '17 at 22:47
• Because you followed the gradients, you followed the path to the true minimum, so you will approximately end up around the ridge solution much of the time. I'm not sure how rigorous you can make this train of thought, but there may be a relationship. – Matthew Drury Jul 19 '17 at 22:48
• @BenoitSanchez This paper might be relevant. The authors tackle a different problem (overfitting in eigenvector computation), but the strategy to deal with overfitting is the same (i.e. implicit regularization by reducing computation). The strategy is to solve a cheaper problem that produces an approximate solution (which - I think - is the same as stopping early in the optimization). – Berk U. Jul 20 '17 at 19:44
• @BenoitSanchez I recommend this. Lorenzo's lectures are available on youtube, but this page also has links to a few papers mit.edu/~9.520/fall17/Classes/early_stopping.html – David Kozak Jul 21 '17 at 6:10

Two alternatives to regularization:

1. Have many, many observations
2. Use a simpler model

Geoff Hinton (co-inventor of back propogation) once told a story of engineers that told him (paraphrasing heavily), "Geoff, we don't need dropout in our deep nets because we have so much data." And his response, was, "Well, then you should build even deeper nets, until you are overfitting, and then use dropout." Good advice aside, you can apparently avoid regularization even with deep nets, so long as there are enough data.

With a fixed number of observations, you can also opt for a simpler model. You probably don't need regularization to estimate an intercept, a slope, and an error variance in a simple linear regression.

Some additional possibilities to avoid overfitting

• Dimensionality reduction

You can use an algorithm such as principal components analysis (PCA) to obtain a lower dimensional features subspace. The idea of PCA is that the variation of your $m$ dimensional feature space may be approximated well by an $l << m$ dimensional subspace.

• Feature selection (also dimensionality reduction)

You could perform a round of feature selection (eg. using LASSO) to obtain a lower dimensional feature space. Something like feature selection using LASSO can be useful if some large but unknown subset of features are irrelevant.

• Use algorithms less prone to overfitting such as random forest. (Depending on the settings, number of features etc..., these can be more computationally expensive than ordinary least squares.)

Some of the other answers have also mentioned the advantages of boosting and bagging techniques/algorithms.

• Bayesian methods

Adding a prior on the coefficient vector an reduce overfitting. This is conceptually related to regularization: eg. ridge regression is a special case of maximum a posteriori estimation.

If you are use a model with a solver, where you can define number of iterations/epochs, you can track validation error and apply early stopping: stop the algorithm, when validation error starts increasing.

• This question clear asks about regression (linear, logistic) models. – Matthew Drury Jul 19 '17 at 22:40
• Technically speaking linear and logistic regression are very simple neural networks. – Andrey Lukyanenko Jul 20 '17 at 3:19
• I don't think that changes my belief that this does not answer the question as asked. If you reworked it to say "if you fit the regression with some form of gradient descent, and applied early stopping" that would be better. – Matthew Drury Jul 20 '17 at 19:31
• Even sklearn has a number of models which support parameter limiting number of iterations. It could be used to track accuracy. But I suppose you are right that the wording isn't exactly correct. – Andrey Lukyanenko Jul 21 '17 at 3:18

Two thoughts:

1. I second the "use a simpler model" strategy proposed by Ben Ogorek.

I work on really sparse linear classification models with small integer coefficients (e.g. max 5 variables with integer coefficients between -5 and 5). The models generalize well in terms of accuracy and trickier performance metrics (e.g calibration).

This method in this paper will scale to large sample sizes for logistic regression, and can be extended to fit other linear classifiers with convex loss functions. It will not handle the cases with lots of features (unless $n/d$ is large enough in which case the data is separable and the classification problem becomes easy).

2. If you can specify additional constraints for your model (e.g. monotonicity constraints, side information), then this can also help with generalization by reducing the hypothesis space (see e.g. this paper).

This needs to be done with care (e.g. you probably want to compare your model to a baseline without constraints, and design your training process in a way that ensures you aren't cherry picking constraints).