Why doesn't regularization solve Deep Neural Nets hunger for data?

Question

An issue I've seen frequently brought up in the context of Neural Networks in general, and Deep Neural Networks in particular, is that they're "data hungry" - that is they don't perform well unless we have a large data set with which to train the network.

My understanding is that this is due to the fact that NNets, especially Deep NNets, have a large number of degrees of freedom. So as a model, a NNet has a very large number of parameters, and if the number of parameters of the model is large relative to the number of training data points, there is an increased tendency to over fit.

But why isn't this issue solved by regularization? As far as I know NNets can use L1 and L2 regularization and also have their own regularization methods like dropout which can reduce the number of parameters in the network.

Can we choose our regularizations methods such that they enforce parsimony and limit the size of the network?

---

To clarify my thinking: Say we are using a large Deep NNet to try to model our data, but the data set is small and could actually be modeled by a linear model. Then why don't the network weights converge in such a way that one neuron simulates the linear regression and all the others converge to zeros? Why doesn't regularization help with this?

Answer 1

The simple way to explain it is that regularization helps to not fit to the noise, it doesn't do much in terms of determining the shape of the signal. If you think of deep learning as a giant glorious function approximator, then you realize that it needs a lot of data to define the shape of the complex signal.

If there was no noise then increasing complexity of NN would produce a better approximation. There would not be any penalty to the size of the NN, bigger would have been better in every case. Consider a Taylor approximation, more terms is always better for non-polynomial function (ignoring numerical precision issues).

This breaks down in presence of a noise, because you start fitting to the noise. So, here comes regularization to help: it may reduce fitting to the noise, thus allowing us to build bigger NN to fit nonlinear problems.

The following discussion is not essential to my answer, but I added in part to answer some comments and motivate the main body of the answer above. Basically, the rest of my answer is like french fires that come with a burger meal, you can skip it.

(Ir)relevant Case: Polynomial regression

Let's look at a toy example of a polynomial regression. It is also a pretty good approximator for many functions. We'll look at the $\sin(x)$ function in $x\in(-3,3)$ region. As you can see from its Taylor series below, 7th order expansion is already a pretty good fit, so we can expect that a polynomial of 7+ order should be a very good fit too:

Next, we're going to fit polynomials with progressively higher order to a small very noisy data set with 7 observations:

We can observe what we've been told about polynomials by many people in-the-know: they're unstable, and start to oscillate wildly with increase in the order of polynomials.

However, the problem is not the polynomials themselves. The problem is the noise. When we fit polynomials to noisy data, part of the fit is to the noise, not to the signal. Here's the same exact polynomials fit to the same data set but with noise completely removed. The fits are great!

Notice a visually perfect fit for order 6. This shouldn't be surprising since 7 observations is all we need to uniquely identify order 6 polynomial, and we saw from Taylor approximation plot above that order 6 is already a very good approximation to $\sin(x)$ in our data range.

Also notice that higher order polynomials do not fit as well as order 6, because there is not enough observations to define them. So, let's look at what happens with 100 observations. On a chart below you see how a larger data set allowed us to fit higher order polynomials, thus accomplishing a better fit!

Great, but the problem is that we usually deal with noisy data. Look at what happens if you fit the same to 100 observations of very noisy data, see the chart below. We're back to square one: higher order polynomials produce horrible oscillating fits. So, increasing data set didn't help that much in increasing the complexity of the model to better explain the data. This is, again, because complex model is fitting better not only to the shape of the signal, but to the shape of the noise too.

Finally, let's try some lame regularization on this problem. The chart below shows regularization (with different penalties) applied to order 9 polynomial regression. Compare this to order (power) 9 polynomial fit above: at an appropriate level of regularization it is possible to fit higher order polynomials to noisy data.

Just in case it wasn't clear: I'm not suggesting to use polynomial regression this way. Polynomials are good for local fits, so a piece-wise polynomial can be a good choice. To fit the entire domain with them is often a bad idea, because they are sensitive to noise, indeed, as it should be evident from plots above. Whether the noise is numerical or from some other source is not that important in this context. the noise is noise, and polynomials will react to it passionately.

Answer 2

At this point in time, its not well-understood when and why certain regularization methods succeed and fail. In fact, its not understood at all why deep learning works in the first place.

Considering the fact that a sufficiently deep neural net can memorize most well-behaved training data perfectly, there are considerably more wrong solutions than there are right for any particular deep net. Regularization, broadly speaking, is an attempt to limit the expressivity of models for these "wrong" solutions - where "wrong" is defined by heuristics we think are important for a particular domain. But often it is difficult to define the heuristic such that you don't lose the "right" expressivity with it. A great example of this is L2 penalties.

Very few methods that would be considered a form of regularization are generally applicable to all application areas of ML. Vision, NLP, and structured prediction problems all have their own cookbook of regularization techniques that have been demonstrated to be effective experimentally for those particular domains. But even within those domains, these techniques are only effective under certain circumstances. For example, batch normalization on deep residual networks appears to make dropout redundant, despite the fact that both have been shown to independently improve generalization.

On a separate note, I think the term regularization is so broad that it makes it difficult to understand anything about it. Considering the fact that convolutions restrict the parameter space exponentially with respect to pixels, you could consider the convolutional neural network a form of regularization on the vanilla neural net.

Answer 3

One class of theorems that show why this problem is fundamental are the No Free Lunch Theorems. For every problem with limited samples where a certain regularization helps, there is another problem where that same regularization will make things worse. As Austin points out, we generally find that L1/L2 regularization are helpful for many real-world problems, but this is only an observation and, because of the NFL theorems, there can be no general guarantees.

Answer 4

I would say that at a high level, the inductive bias of DNNs (deep neural networks) is powerful but slightly too loose or not opinionated enough. By that I mean that DNNs capture a lot of surface statistics about what is going on, but fail to get to the deeper causal/compositional high level structure. (You could view convolutions as a poor's man inductive bias specification).

In addition, it is believed in the machine learning community that the best way to generalize (making good inferences/predictions with little data) is to find the shortest program that gave rise to the data. But program induction/synthesis is hard and we have no good way of doing it efficiently. So instead we rely on a close approximation which is circuit search, and we know how do that with backpropagation. Here, Ilya Sutskever gives an overview of that idea.

---

To illustrate the difference in generalization power of models represented as actual programs vs deep learning models, I'll show the one in this paper: Simulation as an engine of physical scene understanding.

(A) The IPE [intuitive physics engine] model takes inputs (e.g., perception, language, memory, imagery, etc.) that instantiate a distribution over scenes (1), then simulates the effects of physics on the distribution (2), and then aggregates the results for output to other sensorimotor and cognitive faculties (3)

(B) Exp. 1 (Will it fall?) tower stimuli. The tower with the red border is actually delicately balanced, and the other two are the same height, but the blue-bordered one is judged much less likely to fall by the model and people.

(C) Probabilistic IPE model (x axis) vs. human judgment averages (y axis) in Exp. 1. See Fig. S3 for correlations for other values of σ and ϕ. Each point represents one tower (with SEM), and the three colored circles correspond to the three towers in B.

(D) Ground truth (nonprobabilistic) vs. human judgments (Exp. 1). Because it does not represent uncertainty, it cannot capture people’s judgments for a number of our stimuli, such as the red-bordered tower in B. (Note that these cases may be rare in natural scenes, where configurations tend to be more clearly stable or unstable and the IPE would be expected to correlate better with ground truth than it does on our stimuli.)

My point here is that the fit in C is really good, because the model captures the right biases about how humans make physical judgments. This is in big part because it models actual physics (remember that it is a actual physics engine) and can deal with uncertainty.

Now the obvious question is: can you do that with deep learning? This is what Lerer et al did in this work: Learning Physical Intuition of Block Towers by Example

Their model:

Their model is actually pretty good on the task at hand (predicting the number of falling blocks, and even their falling direction)

But it suffers two major drawbacks:

• It needs a huge amount of data to train properly
• In generalizes only in shallow ways: you can transfer to more realistic looking images, add or remove 1 or 2 blocks. But anything beyond that, and the performance goes down catastrophically: add 3 or 4 blocks, change the prediction task...

There was a comparison study done by Tenenbaum's lab about these two approaches: A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding.

Quoting the discussion section:

The performance of CNNs decreases as there are fewer training data. Although AlexNet (not pretrained) performs better with 200,000 training images, it also suffers more from the lack of data, while pretrained AlexNet is able to learn better from a small amount of training images. For our task, both models require around 1,000 images for their performance to be comparable to the IPE model and humans.

CNNs also have limited generalization ability across even small scene variations, such as changing the number of blocks. In contrast, IPE models naturally generalize and capture the ways that human judgment accuracy decreases with the number of blocks in a stack.

Taken together, these results point to something fundamental about human cognition that neural networks (or at least CNNs) are not currently capturing: the existence of a mental model of the world’s causal processes. Causal mental models can be simulated to predict what will happen in qualitatively novel situations, and they do not require vast and diverse training data to generalize broadly, but they are inherently subject to certain kinds of errors (e.g., propagation of uncertainty due to state and dynamics noise) just in virtue of operating by simulation.

Back to the point I want to make: while neural networks are powerful models, they seem to lack the ability to represent causal, compositional and complex structure. And they make up for that by requiring lots of training data.

And back to your question: I would venture that the broad inductive bias and the fact that neural networks do not model causality/compositionality is why they need so much training data. Regularization is not a great fix because of the way they generalize. A better fix would be to change their bias, as is currently being tried by Hinton with capsules for modelling whole/part geometry, or interaction networks for modelling relations.

Answer 5

To clarify my thinking: Say we are using a large Deep NNet to try to model our data, but the data set is small and could actually be modeled by a linear model. Then why don't the network weights converge in such a way that one neuron simulates the linear regression and all the others converge to zeros? Why doesn't regularization help with this?

Neural nets can be trained like this. If proper L1 regularization used then much of the weights can be zeroed and this will make neural nets behave like concatenation of 1 or so linear regression neurons and many other zero nerons. So yes - L1/L2 regularizations or like that can be used to restrict the size or representational power of the neural network.

Actually the size of the model itself is a kind of regularization - if you make model large, it means that you injects a prior knowledge about the problem, that is, the problems is highly complex so it requires model that have high representational power. If you make model small, then it means you injects knowledge that the problem is simple so model don't need much capacity.

And this means L2 regularization will not make networks "sparse" as you described, because L2 regularization injects prior knowledge that contribution of each neuron (weight) should be small but non-zero. So network would use each of the neurons rather than use only small set of neurons.

Answer 6

First of all there are plenty of regularization methods both in use and in active research for deep learning. So your premise isn't entirely certain.

As for methods in use, weight decay is a direct implementation of an L2 penalty on the weights via gradient descent. Take the gradient of the squared norm of your weights and add a small step in this direction to them at each iteration. Dropout is also considered a form of regularization, which imposes a kind of averaged structure. This would seem to imply something like an L2 penalty over an ensemble of networks with shared parameters.

You could presumably crank up the level of these or other techniques to address small samples. But note that regularization implies imposition of prior knowledge. The L2 penalty on the weights implies a Gaussian prior for the weights, for example. Increasing the amount of regularization essentially states that your prior knowledge is increasingly certain and biases your result towards that prior. So you can do it and it will overfit less but the biased output may suck. Obviously the solution is better prior knowledge. For image recognition this would mean much more structured priors regarding the statistics of your problem. The problem with this direction is you are imposing lots of domain expertise, and avoiding having to impose human expertise was one of the reasons you used deep learning.

Answer 7

Regularization is a method for including prior information into a model. This will seem straightforward from the Bayesian perspective but is easy to see outside from the perspective as well. For example, the $L_2$ penalty + standarization of covariates in Ridge Regression is essentially using the prior information that we don't believe that estimation should be entirely dominated by a small number of predictors. Similarly, the $L_1$ penalty can be seen as "betting on sparseness of the solution" (side note: this doesn't make sense from the traditional Bayesian perspective but that's another story...).

A key point here is that regularization isn't always helpful. Rather, regularizing toward what should probably be true is very helpful, but regularizing in the wrong direction is clearly bad.

Now, when it comes to deep neural nets, the interpretability of this models makes regularization a little more difficult. For example, if we're trying to identify cats, in advance we know that "pointy ears" is an important feature. If we were using some like logistic regression with an $L_2$ penalty and we had an indicator variable "pointy ears" in our dataset, we could just reduce the penalty on the pointy ears variable (or better yet, penalize towards a positive value rather than 0) and then our model would need less data for accurate predictions.

But now suppose our data is images of cats fed into a deep neural networks. If "pointy ears" is, in fact, very helpful for identifying cats, maybe we would like to reduce the penalty to give this more predictive power. But we have no idea where in the network this will be represented! We can still introduce penalties so that some small part of the system doesn't dominate the whole network, but outside of that, it's hard to introduce regularization in a meaningful way.

In summary, it's extremely difficult to incorporate prior information into a system we don't understand.