70
$\begingroup$

Unlike other articles, I found the wikipedia entry for this subject unreadable for a non-math person (like me).

I understood the basic idea, that you favor models with fewer rules. What I don't get is how do you get from a set of rules to a 'regularization score' which you can use to sort the models from least to most overfit.

Can you describe a simple regularization method?

I'm interested in the context of analyzing statistical trading systems. It would be great if you could describe if/how I can apply regularization to analyze the following two predictive models:

Model 1 - price going up when:

  • exp_moving_avg(price, period=50) > exp_moving_avg(price, period=200)

Model 2 - price going up when:

  • price[n] < price[n-1] 10 times in a row
  • exp_moving_avg(price, period=200) going up

But I'm more interested in getting a feeling for how you do regularization. So if you know better models for explaining it please do.

$\endgroup$
  • 7
    $\begingroup$ An example is ridge regression, which is OLS with a bound on the sum of the squared coefficients. This will introduce bias into the model, but will reduce the variance of the coefficients, sometimes substantially. LASSO is another related method, but puts an L1 constraint on the size of the coefficients. It has the advantage of dropping coefficients. This is useful for p>>n situations Regularizing, in a way, means "shrinking" the model to avoid over-fitting (and to reduce coefficient variance), which usually improves the model's predictive performance. $\endgroup$ – HairyBeast Nov 27 '10 at 18:30
  • 2
    $\begingroup$ @HairyBeast You should put your nice comment as an answer. If possible, try to add an illustrative example so that the OP can figure out how it translates to the problem at hand. $\endgroup$ – chl Nov 28 '10 at 12:02
  • 1
    $\begingroup$ @HairyBeast, so can I say that regularization is just a method to implement the idea of bias-variance tradeoff? $\endgroup$ – avocado Dec 24 '13 at 14:00
  • $\begingroup$ I found this video very helpful, particularly in visualizing the different forms of Lp regularization: youtube.com/watch?v=sO4ZirJh9ds $\endgroup$ – Anm Aug 19 '16 at 16:12
  • $\begingroup$ Regularization is for addressing the overfit in the model that is learnt. Tried to explain in plain English and visually. Following is the link to the article medium.com/@vamsi149/… $\endgroup$ – solver149 Aug 30 '18 at 22:49
55
$\begingroup$

In simple terms, regularization is tuning or selecting the preferred level of model complexity so your models are better at predicting (generalizing). If you don't do this your models may be too complex and overfit or too simple and underfit, either way giving poor predictions.

If you least-squares fit a complex model to a small set of training data you will probably overfit, this is the most common situation. The optimal complexity of the model depends on the sort of process you are modeling and the quality of the data, so there is no a-priori correct complexity of a model.

To regularize you need 2 things:

  1. A way of testing how good your models are at prediction, for example using cross-validation or a set of validation data (you can't use the fitting error for this).
  2. A tuning parameter which lets you change the complexity or smoothness of the model, or a selection of models of differing complexity/smoothness.
Basically you adjust the complexity parameter (or change the model) and find the value which gives the best model predictions.

Note that the optimized regularization error will not be an accurate estimate of the overall prediction error so after regularization you will finally have to use an additional validation dataset or perform some additional statistical analysis to get an unbiased prediction error.

An alternative to using (cross-)validation testing is to use Bayesian Priors or other methods to penalize complexity or non-smoothness, but these require more statistical sophistication and knowledge of the problem and model features.

$\endgroup$
  • 2
    $\begingroup$ +1 from me. I like that this answer is starting at the beginning and so easy to understand... $\endgroup$ – Andrew Nov 22 '11 at 14:40
  • 2
    $\begingroup$ Is regularization really ever used to reduce underfitting? In my experience, regularization is applied on a complex/sensitive model to reduce complexity/sensitvity, but never on a simple/insensitive model to increase complexity/sensitivity. $\endgroup$ – Richard Hardy Mar 23 '17 at 9:43
  • $\begingroup$ This answer is now quite old, but I assume what Toby was referring to is that regularization is a principled way to fit a model of appropriate complexity given the amount of data; it is an alternative both to selecting a priori a model with too few parameters (or the wrong ones), and also to selecting a model that is too complex and overfits. $\endgroup$ – Bryan Krause Aug 30 '18 at 23:06
26
$\begingroup$

Suppose you perform learning via empirical risk minimization.

More precisely:

  • you have got your non-negative loss function $L(\text{actual value},\text{ predicted value})$ which characterize how bad your predictions are
  • you want to fit your model in a such way that its predictions minimize mean of loss function, calculated only on training data (the only data you have)

Then the aim of learning process is to find $\text{Model} = \text{argmin} \sum L(\text{actual}, \text{predicted}(\text{Model}))$ (this method is called empirical risk minimization).

But if you haven't got enough data and there is a huge amount of variables in your model, it is very probable to find such a model that not only explain patterns but also explains random noise in your data. This effect is called overfitting and it leads to degradation of generalization ability of your model.

In order to avoid overfitting a regularization term is introduced into the target function: $\text{Model} = \text{argmin} \sum L(\text{actual}, \text{predicted}(\text{Model})) + \lambda R(\text{Model})$

Usually, this term $R(\text{Model})$ imposes a special penalty on complex models. For instance, on models with large coefficients (L2 regularization, $R$=sum of squares of coefficients) or with a lot if non-zero coefficients (L1 regularization, $R$=sum of absolute values of coefficients). If we are training decision tree, $R$ can be its depth.

Another point of view is that $R$ introduces our prior knowledge about a form of the best model ("it doesn't have too large coefficients", "it is almost orthogonal to $\bar a$")

$\endgroup$
12
$\begingroup$

Put in simple terms, regularization is about benefiting the solutions you'd expect to get. As you mention, for example you can benefit "simple" solutions, for some definition of simplicity. If your problem has rules, one definition can be fewer rules. But this is problem-dependent.

You're asking the right question, however. For example in Support Vector Machines this "simplicity" comes from breaking ties in the direction of "maximum margin". This margin is something that can be clearly defined in terms of the problem. There is a very good geometric derivation in the SVM article in Wikipedia. It turns out that the regularization term is, arguably at least, the "secret sauce" of SVMs.

How do you do regularization? In general that comes with the method you use, if you use SVMs you're doing L2 regularization, if your using LASSO you're doing L1 regularization (see what hairybeast is saying). However, if you're developing your own method, you need to know how to tell desirable solutions from non-desirable ones, and have a function that quantifies this. In the end you'll have a cost term and a regularization term, and you want to optimize the sum of both.

$\endgroup$
1
$\begingroup$

Regularization techniques are techniques applied to machine learning models which make the decision boundary / fitted model smoother. Those techniques help to prevent overfitting.

Examples: L1, L2, Dropout, Weight Decay in Neural Networks. Parameter $C$ in SVMs.

$\endgroup$
0
$\begingroup$

In simple term, Regularization is a technique to avoid over-fitting when training machine learning algorithms. If you have an algorithm with enough free parameters you can interpolate with great detail your sample, but examples coming outside the sample might not follow this detail interpolation as it just captured noise or random irregularities in the sample instead of the true trend.

Over-fitting is avoided by limiting the absolute value of the parameters in the model.This can be done by adding a term to the cost function that imposes a penalty based on the magnitude of the model parameters. If the magnitude is measured in L1 norm this is called "L1 regularization" (and usually results in sparse models), if it is measured in L2 norm this is called "L2 regularization", and so on.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.