I am hearing these words more and more as I study machine learning. In fact, some people have won Fields medal working on regularities of equations. So, I guess this is a term that carries itself from statistical physics/maths to machine learning. Naturally, a number of people I asked just couldn't intuitively explain it.

I know that methods such as dropout help in regularization (=> they say it reduces overfitting, but I really don't get what it is: if it only reduces overfitting, why not just call it anti-overfitting methods => there must be something more I think, hence this question).

I would be really grateful (I guess the naive ML community would be too!) if you could explain:

  1. How do you define regularity? What is regularity?

  2. Is regularization a way to ensure regularity? i.e. capturing regularities?

  3. Why do ensembling methods like dropout, normalization methods all claim to be doing regularization?

  4. Why do these (regularity/regularization) come up in machine learning?

Thanks a lot for your help.


Regularization is employed in almost all machine learning algorithms where we're trying to learn from finite samples of training data.

I'll attempt to indirectly answer your specific questions by explaining the genesis of the concept of regularization. The full theory is much more detailed and this explanation should not be interpreted as complete, but its intended to simply point you in the right direction for further exploration. Since your primary objective is to get an intuitive understanding of regularization, I've summarized and heavily simplified the following explanation from Chapter 7 of "Neural Networks and Learning Machines", 3rd edition by Simon Haykin (and omitted several details while doing so).

Lets revisit the supervised learning problem with independent variables $x_i$ and dependent variable $y_i$ as trying to find a function $f$ that will be able to "map" the input X to an output Y.

To take this further, lets understand Hadamard's terminology of a "well-posed" problem - a problem is well-posed if it satisfies the following three conditions:

  1. For every input $x_i$, and output $y_i$ exists.
  2. For a pair of inputs $x_1$ and $x_2$, $f(x_1) = f(x_2)$ if and only if $x_1 = x_2$.
  3. The mapping $f$ is continuous (stability criteria)

For supervised learning, these conditions may be violated since:

  1. A distinct output may not exist for a given input.
  2. There may not be enough information in the training samples to construct a unique input-output mapping (since running the learning algorithm on different training samples results in different mapping functions).
  3. Noise in the data adds uncertainty to the reconstruction process which may effect its stability.

For solving such "ill-posed" problems, Tikhonov proposed a regularization method to stabilize the solution by including a non-negative functional that embeds prior information about the solution.

The most common form of prior information involves the assumption that the input-output mapping function is smooth - i.e. similar inputs produce similar outputs.

Tikhnov's regularization theory adds the regularization term to the cost function (loss function to be minimized) which includes the regularization parameter $\lambda$ and the assumed form of the mapping $f$. The value of $\lambda$ is chosen between 0 and $\infty$. A value of 0 implies the solution is determined completely from the training samples; whereas a value of $\infty$ implies the training examples are unreliable.

So the regularization parameter $\lambda$ is selected and optimized to achieve the desired balance between model bias and model variance by incorporating the right amount of prior information into it.

Some examples of such regularized cost functions are:

Linear Regression:

$ J(\theta) = \frac 1m \sum_{i=1}^m [ h_\theta(x^i) - y^i]^2 + \frac \lambda{2m} \sum_{j=1}^n \theta_j^2 $

Logistic Regression:

$ J(\theta) = \frac 1m \sum_{i=1}^m [ -y^i log(h_\theta(x^i)) - (1-y^i)log(1 - h_\theta(x^i))] + \frac \lambda{2m} \sum_{j=1}^n \theta_j^2 $

Where, $\theta$ are the coefficients we've identified for $x$ , and $h_\theta(x)$ is the estimate of $y$ .

The second summation term in each example is the regularization term. Since this term is always a non-negative value, it stops the optimizer from reaching the global minima for the cost function. The form of the term shown here is an $L_2$ regularization. There are many variations in the form of the regularization function, the commonly used forms are: lasso, elastic net and ridge regression. These have their own advantages and disadvantages which help decide where their best applicability.

The net effect of applying regularization is to reduce model complexity which reduces over-fitting. Other approaches to regularization (not listed in the examples above) include modifications to structural models such as regression/classification Trees, boosted trees, etc. by dropping out nodes to make simpler trees. More recently this has been applied in so-called "deep learning" by dropping out connections between neurons in a neural network.

A specific answer to Q3 is that some ensembling methods such as Random Forest (or similar voting schemes) achieve regularization due to their inherent method, i.e. voting and electing the response from a collection of un-regularized Trees. Even though the individual trees have overfit, the process of "averaging out" their outcome stops the ensemble from overfitting to the training set.


The concept of regularity belongs to axiomatic set theory, you could refer to this article for pointers - en.wikipedia.org/wiki/Axiom_of_regularity and explore this topic further if you're interested in the details.

On regularization for neural nets: When adjusting the weights while running the back-propagation algorithm, the regularization term is added to the cost function in the same manner as the examples for linear and logistic regression. So the addition of the regularization term stops the back-propagation from reaching the global minima.

The article describing batch normalization for neural networks is - Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe, Szegedy, 2015. Its been known that backpropagation to train a neural network works better when the input variables are normalized. In this paper, the authors have applied normalization to each mini-batch used in Stochastic Gradient Descent to avoid the problem of "vanishing gradients" when training many layers of a neural network. The algorithm described in their paper treats the mean and variance computed in each batch for each layer of activations as another set of parameters optimized in mini-batch SGD (in addition to the NN weights). The activations are then normalized using the entire training set. You may refer to their paper for full details of this algorithm. By using this method, they were able to avoid using dropouts for regularization, and hence their claim that this is another type of regularization.

  • $\begingroup$ thanks for the great answer. Could you mathematically explain a bit how methods such as normalization achieve regularization? In a talk by Goodfellow, he said that anything that is differentiable can act as a regularizer for a neural net. Also, do you know what regularities are? do they just mean patterns or is there some maths behind that? thanks again. $\endgroup$ – Rafael Feb 14 '17 at 9:35
  • $\begingroup$ Thanks for the reply. I can't remember the talk. In neural nets we add layers such as batch normalization. I wanted to know hoe they contribute to regularization? $\endgroup$ – Rafael Feb 16 '17 at 9:10
  • $\begingroup$ Edited to answer your comment as well as add back answers given in the earlier comments. $\endgroup$ – Sandeep S. Sandhu Feb 16 '17 at 16:53

Question 1

I am not aware of any canonical definition, and your questions suggests that this term is used with different meanings. Let's start with simple examples (which will answer question 2).

Question 2

The ridge regression may be a good starting point. It is a regularization method that circumvent the issue raised by a singular matrix.

However, the "regularization parameter" defined in gradient boosting methods (per example) is here to ensure a low complexity for the model.

Question 3

Normalization as regularization has another meaning (and this terminology is quite misleading). It turns a complex problem "from the gradient descent point of view" into something simpler. Though it is not needed to calibrate a neural network, it really helps during the calibration. (However, note that if we could find the global extrema of arbitrary functions, normalization would not be needed)

Question 4

Regularization (as a way to reduce the complexity of a model) is used to reduce overfit. The less complex a model is, the less likely it is to overfit.


S. Watanabe makes a rigorous usage on this terminology in his research.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.