Regularization in regression (linear, logistic...) is the most popular way to reduce over-fitting.

When the goal is prediction accuracy (not explaining), are there any good alternatives to regularization, especially suited for big data-sets (mi/billions of observations and millions of features)?

  • 3
    $\begingroup$ "Big datasets" may mean a lot of observations, a lot of variables or both, and the answer may depend on the number of observations and variables. $\endgroup$
    – Pere
    Commented Jul 19, 2017 at 10:45
  • $\begingroup$ Why not use norm regularisation? For neural networks , there is dropout $\endgroup$
    – seanv507
    Commented Jul 19, 2017 at 12:05
  • 4
    $\begingroup$ The advantage of regularization is that it's computationally cheap. Ensemble methods such as bagging and boosting (etc.) combined with cross validation methods for model diagnostics are a good alternative, but it will be a much more costly solution. $\endgroup$
    – Digio
    Commented Jul 19, 2017 at 14:04
  • 1
    $\begingroup$ This might be of interest: stats.stackexchange.com/a/161592/40604 $\endgroup$
    – Dan
    Commented Jul 19, 2017 at 22:51
  • 1
    $\begingroup$ To add to the comment by Digio: regularization is cheap compared to bagging/boosting but still expensive compared to the alternative of "no regularization" (see e.g. this post by Ben Recht on how regularization makes deep learning hard). If you have a huge number of samples, no regularization can work well for far cheaper. The model can still generalize well as @hxd1001 points out) $\endgroup$
    – Berk U.
    Commented Jul 20, 2017 at 19:34

7 Answers 7


Two important points that are not directly related to your question:

  • First, even the goal is accuracy instead of interpretation, regularization is still necessary in many cases, since, it will make sure the "high accuracy" on real testing / production data set, not the data used for modeling.

  • Second, if there are billion rows and million columns, it is possible no regularization is needed. This is because the data is huge, and many computational models have "limited power", i.e., it is almost impossible to overfit. This is why some deep neural network has billions of parameters.

Now, about your question. As mentioned by Ben and Andrey, there are some options as alternatives to regularization. I would like to add more examples.

  • Use simpler model (For example, reduce number of hidden unit in neural network. Use lower order polynomial kernel in SVM. Reduce number of Gaussians in mixture of Gaussian. etc.)

  • Stop early in the optimization. (For example, reduce the epoch in neural network training, reduce number of iterations in optimization (CG, BFGS, etc.)

  • Average on many models (For example, random forest etc.)

  • $\begingroup$ Thanks a lot. The second option (stop early) is what we are trying presently with SGD. It works rather well. We want to compare it with regularization soon. Are you aware of any article that mentions this method ? $\endgroup$ Commented Jul 19, 2017 at 19:47
  • 1
    $\begingroup$ There is a hint of a geometric relationship between early stopping with gradient descent, and regularization. For example, ridge regression in it's primal form asks for the parameters minimizing the loss function that lie within a solid ellipse centered at the origin, with the size of the ellipse a function of the regularization strength. The ridge parameters lie on the surface of the ellipse if they are different than the un-regularized solution. If you run an ascent starting at the origin, and then stop early, you will be on the boundary of one of these ellipses... $\endgroup$ Commented Jul 19, 2017 at 22:47
  • $\begingroup$ Because you followed the gradients, you followed the path to the true minimum, so you will approximately end up around the ridge solution much of the time. I'm not sure how rigorous you can make this train of thought, but there may be a relationship. $\endgroup$ Commented Jul 19, 2017 at 22:48
  • 1
    $\begingroup$ @BenoitSanchez This paper might be relevant. The authors tackle a different problem (overfitting in eigenvector computation), but the strategy to deal with overfitting is the same (i.e. implicit regularization by reducing computation). The strategy is to solve a cheaper problem that produces an approximate solution (which - I think - is the same as stopping early in the optimization). $\endgroup$
    – Berk U.
    Commented Jul 20, 2017 at 19:44
  • $\begingroup$ @BenoitSanchez I recommend this. Lorenzo's lectures are available on youtube, but this page also has links to a few papers mit.edu/~9.520/fall17/Classes/early_stopping.html $\endgroup$ Commented Jul 21, 2017 at 6:10

Two alternatives to regularization:

  1. Have many, many observations
  2. Use a simpler model

Geoff Hinton (co-inventor of back propogation) once told a story of engineers that told him (paraphrasing heavily), "Geoff, we don't need dropout in our deep nets because we have so much data." And his response, was, "Well, then you should build even deeper nets, until you are overfitting, and then use dropout." Good advice aside, you can apparently avoid regularization even with deep nets, so long as there are enough data.

With a fixed number of observations, you can also opt for a simpler model. You probably don't need regularization to estimate an intercept, a slope, and an error variance in a simple linear regression.

  • $\begingroup$ I still consider at least L1-regularization useful against outliers... of course, big data can smooth outliers' effect but we rarely have big data in real life... and your advice about Dropout , anyway, is exactly about the Regularization, but not its absence $\endgroup$
    – JeeyCi
    Commented Mar 10 at 12:07
  • $\begingroup$ L2-regularization against multicollinearity - while not preventing predictions, still can induce disturbance in statistical scores of estimator $\endgroup$
    – JeeyCi
    Commented Mar 10 at 12:12
  • 1
    $\begingroup$ @JeeyCi, more data can help with multicollinearity, since it's a low information problem. $\endgroup$
    – Ben Ogorek
    Commented Mar 11 at 13:49
  • 1
    $\begingroup$ 3rd alternative is early_stopping (will not re-read the whole page to find if somebody mentioned it) $\endgroup$
    – JeeyCi
    Commented Mar 15 at 17:29

Some additional possibilities to avoid overfitting

  • Dimensionality reduction

    You can use an algorithm such as principal components analysis (PCA) to obtain a lower dimensional features subspace. The idea of PCA is that the variation of your $m$ dimensional feature space may be approximated well by an $l << m$ dimensional subspace.

  • Feature selection (also dimensionality reduction)

    You could perform a round of feature selection (eg. using LASSO) to obtain a lower dimensional feature space. Something like feature selection using LASSO can be useful if some large but unknown subset of features are irrelevant.

  • Use algorithms less prone to overfitting such as random forest. (Depending on the settings, number of features etc..., these can be more computationally expensive than ordinary least squares.)

    Some of the other answers have also mentioned the advantages of boosting and bagging techniques/algorithms.

  • Bayesian methods

    Adding a prior on the coefficient vector an reduce overfitting. This is conceptually related to regularization: eg. ridge regression is a special case of maximum a posteriori estimation.

  • $\begingroup$ Maximum Likelihood Estimator (with IRLS, or Gradient Descent) still is prone to overfitting without standard statistical approaches for Generalization/Approximation purposes, though MLE is still helpful in non-normally distributed data approximation or separation (in discriminant analysis) $\endgroup$
    – JeeyCi
    Commented Mar 10 at 12:16

Two thoughts:

  1. I second the "use a simpler model" strategy proposed by Ben Ogorek.

    I work on really sparse linear classification models with small integer coefficients (e.g. max 5 variables with integer coefficients between -5 and 5). The models generalize well in terms of accuracy and trickier performance metrics (e.g calibration).

    This method in this paper will scale to large sample sizes for logistic regression, and can be extended to fit other linear classifiers with convex loss functions. It will not handle the cases with lots of features (unless $n/d$ is large enough in which case the data is separable and the classification problem becomes easy).

  2. If you can specify additional constraints for your model (e.g. monotonicity constraints, side information), then this can also help with generalization by reducing the hypothesis space (see e.g. this paper).

    This needs to be done with care (e.g. you probably want to compare your model to a baseline without constraints, and design your training process in a way that ensures you aren't cherry picking constraints).


If you are use a model with a solver, where you can define number of iterations/epochs, you can track validation error and apply early stopping: stop the algorithm, when validation error starts increasing.

  • 1
    $\begingroup$ This question clear asks about regression (linear, logistic) models. $\endgroup$ Commented Jul 19, 2017 at 22:40
  • 2
    $\begingroup$ Technically speaking linear and logistic regression are very simple neural networks. $\endgroup$ Commented Jul 20, 2017 at 3:19
  • 2
    $\begingroup$ I don't think that changes my belief that this does not answer the question as asked. If you reworked it to say "if you fit the regression with some form of gradient descent, and applied early stopping" that would be better. $\endgroup$ Commented Jul 20, 2017 at 19:31
  • $\begingroup$ Even sklearn has a number of models which support parameter limiting number of iterations. It could be used to track accuracy. But I suppose you are right that the wording isn't exactly correct. $\endgroup$ Commented Jul 21, 2017 at 3:18
  • $\begingroup$ @ Matthew Drury: can use momentum Stochastic Gradient Descent & stop according tolerance param $\endgroup$
    – JeeyCi
    Commented Mar 10 at 12:22

What is regularization, really?

Perhaps you are conflating L1/L2 regularization (aka. Lasso/ridge regression, Tikhonov regularization...), the most ubiquitous type, as the only type of regularization 🤔

Regularization is actually anything that prevents overfitting, that you can do to a learning algorithm [Wikipedia]. Dropout, batch normalization, early stopping, model ensembling, feature selection, and many of the techniques others have pointed out here... are all just different regularization techniques!

Spiderman meme: L1/L2, ensembling, early stopping are all regularization.

Bias-variance tradeoff

Perhaps thinking about this issue in terms of the bias-variance tradeoff, a fundamental machine learning concept, could greatly clarify your thoughts.

If our goal is prediction accuracy, we want to reduce the expected error of a supervised learner $\hat{f}$, which can be decomposed into bias, variance, and irreducible error:

$$ {\displaystyle \operatorname {E} _{D}{\Big [}{\big (}y-{\hat {f}}(x;D){\big )}^{2}{\Big ]}={\Big (}\operatorname {Bias} _{D}{\big [}{\hat {f}}(x;D){\big ]}{\Big )}^{2}+\operatorname {Var} _{D}{\big [}{\hat {f}}(x;D){\big ]}+\sigma ^{2}} $$

Regularization penalizes complex models, to try to reduce the variance of the estimator (more than the bias is increased), to ultimately reduce the expected error. Philosophically, this is akin to Occam's razor, where we introduce an inductive bias for simplicity on the assumption that "simpler is better".

We usually want to regularize

From a Bayesian viewpoint, we can also show that including L1/L2 regularization means placing a prior and obtaining a MAP estimate, instead of an MLE estimate (see here).

Overfitting is simply when your model is unable to generalize well to your actual data of interest ("test" or "production" dataset), usually because it has fit to your training data too well. We always want to prevent this with some form of regularization.

  • $\begingroup$ +1 for 'Bias-variance tradeoff' concept, but "simpler is better" is doubtful for me, the BLUE is better for OLS ( Best Linear Unbiased Estimations: unbiased, efficient, consistent). Still bias of demeaned data sometimes is informative, e,g, in AB-testing $\endgroup$
    – JeeyCi
    Commented Mar 10 at 12:29

Other alternatives to regularization:

  • Using ensemble methods.
  • Oversampling and data augmentation.
  • Combine the variables to get new ones, for example with PCA.
  • Adding random noise at every step of the optimization.
  • Smoothing the data.
  • Dropout: typically used with NN, but could also be used with the covariates.
  • Standardization always improves the results.
  • Using a bayesian prior is equivalent to regularization.
  • Replace categorical (with many values) fixed effects with random effects (because it reduces the number of parameters).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.