What is regularization in plain english?

Question

Unlike other articles, I found the wikipedia entry for this subject unreadable for a non-math person (like me).

I understood the basic idea, that you favor models with fewer rules. What I don't get is how do you get from a set of rules to a 'regularization score' which you can use to sort the models from least to most overfit.

Can you describe a simple regularization method?

I'm interested in the context of analyzing statistical trading systems. It would be great if you could describe if/how I can apply regularization to analyze the following two predictive models:

Model 1 - price going up when:

• exp_moving_avg(price, period=50) > exp_moving_avg(price, period=200)

Model 2 - price going up when:

• price[n] < price[n-1] 10 times in a row
• exp_moving_avg(price, period=200) going up

But I'm more interested in getting a feeling for how you do regularization. So if you know better models for explaining it please do.

Answer 1

In simple terms, regularization is tuning or selecting the preferred level of model complexity so your models are better at predicting (generalizing). If you don't do this your models may be too complex and overfit or too simple and underfit, either way giving poor predictions.

If you least-squares fit a complex model to a small set of training data you will probably overfit, this is the most common situation. The optimal complexity of the model depends on the sort of process you are modeling and the quality of the data, so there is no a-priori correct complexity of a model.

To regularize you need 2 things:

1. A way of testing how good your models are at prediction, for example using cross-validation or a set of validation data (you can't use the fitting error for this).
2. A tuning parameter which lets you change the complexity or smoothness of the model, or a selection of models of differing complexity/smoothness.

Basically you adjust the complexity parameter (or change the model) and find the value which gives the best model predictions.

Note that the optimized regularization error will not be an accurate estimate of the overall prediction error so after regularization you will finally have to use an additional validation dataset or perform some additional statistical analysis to get an unbiased prediction error.

An alternative to using (cross-)validation testing is to use Bayesian Priors or other methods to penalize complexity or non-smoothness, but these require more statistical sophistication and knowledge of the problem and model features.

Answer 2

Suppose you perform learning via empirical risk minimization.

More precisely:

• you have got your non-negative loss function $L(\text{actual value},\text{ predicted value})$ which characterize how bad your predictions are
• you want to fit your model in a such way that its predictions minimize mean of loss function, calculated only on training data (the only data you have)

Then the aim of learning process is to find $\text{Model} = \text{argmin} \sum L(\text{actual}, \text{predicted}(\text{Model}))$ (this method is called empirical risk minimization).

But if you haven't got enough data and there is a huge amount of variables in your model, it is very probable to find such a model that not only explain patterns but also explains random noise in your data. This effect is called overfitting and it leads to degradation of generalization ability of your model.

In order to avoid overfitting a regularization term is introduced into the target function: $\text{Model} = \text{argmin} \sum L(\text{actual}, \text{predicted}(\text{Model})) + \lambda R(\text{Model})$

Usually, this term $R(\text{Model})$ imposes a special penalty on complex models. For instance, on models with large coefficients (L2 regularization, $R$=sum of squares of coefficients) or with a lot if non-zero coefficients (L1 regularization, $R$=sum of absolute values of coefficients). If we are training decision tree, $R$ can be its depth.

Another point of view is that $R$ introduces our prior knowledge about a form of the best model ("it doesn't have too large coefficients", "it is almost orthogonal to $\bar a$")

Answer 3

Put in simple terms, regularization is about benefiting the solutions you'd expect to get. As you mention, for example you can benefit "simple" solutions, for some definition of simplicity. If your problem has rules, one definition can be fewer rules. But this is problem-dependent.

You're asking the right question, however. For example in Support Vector Machines this "simplicity" comes from breaking ties in the direction of "maximum margin". This margin is something that can be clearly defined in terms of the problem. There is a very good geometric derivation in the SVM article in Wikipedia. It turns out that the regularization term is, arguably at least, the "secret sauce" of SVMs.

How do you do regularization? In general that comes with the method you use, if you use SVMs you're doing L2 regularization, if your using LASSO you're doing L1 regularization (see what hairybeast is saying). However, if you're developing your own method, you need to know how to tell desirable solutions from non-desirable ones, and have a function that quantifies this. In the end you'll have a cost term and a regularization term, and you want to optimize the sum of both.

Answer 4

Regularization techniques are techniques applied to machine learning models which make the decision boundary / fitted model smoother. Those techniques help to prevent overfitting.

Examples: L1, L2, Dropout, Weight Decay in Neural Networks. Parameter $C$ in SVMs.

Answer 5

In simple term, Regularization is a technique to avoid over-fitting when training machine learning algorithms. If you have an algorithm with enough free parameters you can interpolate with great detail your sample, but examples coming outside the sample might not follow this detail interpolation as it just captured noise or random irregularities in the sample instead of the true trend.

Over-fitting is avoided by limiting the absolute value of the parameters in the model.This can be done by adding a term to the cost function that imposes a penalty based on the magnitude of the model parameters. If the magnitude is measured in L1 norm this is called "L1 regularization" (and usually results in sparse models), if it is measured in L2 norm this is called "L2 regularization", and so on.