Two questions about standardization and overfitting [closed]

Question

Question 1

Why neural networks (or more generally, any machine learning models) tend to overfit smaller datasets?

• The "default" reason is that the information associated with the smaller dataset is not sufficient for models like neural network (which has large capacity) to learn. For example, if there are just a few images available, then the parameters of neural network are barely updated and nothing much is learnt from them.

• Another reason may come statistical learning theory, which shows that some networks's error grow with order $O(\frac{1}{\sqrt{m}})$, where $m$ is number of examples.

However, I am not quite satisfied with these reasons. The first one is intuitive enough but lacks quantitative reasoning and latter one is limited in particular type of neural network architecture.

Update

Hopefully the following two resources could provide more context for my question.

• A guide to choose suitable machine learning model by scikit-learn (original image could be found here). Based on range of the amount of data available (less than 50, 50 - 100000, etc), it provides different paths to choose suitable models. As much as I know there numbers are largely empirical, I am wondering what is the theory under the hood.
• The following is a snapshot from Andrew Ng's deep learning course. When amount of data is small, the performance of neural network may not be as good as traditional machine learning algorithms.

Question 2

In computer vision tasks, why it is a general practice that images are just scaled to $[0, 1]$ while normalization $x_i \leftarrow \frac{x_i-\mu_i}{\sigma_i}$ is preferable for optimization purposes? ($x_i$ is individual pixel value at location $i$).

The only reason I could think of is convenience. Are there deep reasons behind this practice. Say, is there any quantitative rationale that pinpoint their differences?

Any input is appreciated. Thank you in advance!

Answer 1

Question 1

Regarding model capacity

I'm going to go with the first reason you state and explain it a bit better. There should a sort of balance between model capacity and dataset size. If the capacity of a model is much higher than necessary$^1$ for a particular dataset, then that model will overfit to it.

The extreme case is when a model's capacity is so high that it can completely memorize the training data. This means that the model has learned the data so well that it achieves zero error. Obviously, for a given network, the smaller the dataset, the easier it is for a network to memorize.

Now, because neural networks typically have a large number of parameters (i.e. high capacity), it is very easy for them to memorize small datasets. Furthermore, even if it can't completely memorize the data, it will certainly overfit on it.

On the other hand, if we have a large number of samples, it will be very hard for the model to memorize them. In order to improve its performance, the network will actually have to identify the relevant information and model it.

Thus the problem isn't as you state that the network can't learn from small datasets, but that it's too easy to learn from them.

$^1$ This is pretty arbitrary. The necessary capacity is one that allows the model to learn all useful relationships between the data and their labels but not the underlying noise in the data.

Regarding dimensionality

Another way to view this is through the lens of dimensionality. Say you have a very simple problem to solve (e.g. the XOR problem). You can actually need $4$ samples to solve this. Why? because it's a very low-dimensional and easy task.

However as the dimensionality of the data increases you need a much higher number of samples to sufficiently represent the feature space.

Given the same number of samples, as the dimensionality of the data increases, they become exponentially sparser. This causes the data to be easily separable by a classifier (i.e. in the figure above it's much higher to distinguish between green and red dots in the ligh-dimensional space). This is up to a certain point a good thing because its easier to learn. However, it's also much easier to overfit on high-dimensional datasets!

For example, say you want to classify $256\times256$ RGB images. This data has a dimensionality of $256 \cdot 256 \cdot 3 = 196608$. To properly learn on this data you need a large number of samples, or else the data won't sufficiently represent the vast feature space and high-capacity models can overfit with ease on this dataset.

Summing up...

If you have few samples distributed in a large feature space, then it's easy for a model to memorize them (or at the least overfit). The higher the capacity of a model, the easier it is for it to overfit.

So if you have a high dimensionality and a high-capacity model you need a lot of data.

---

Question 2

When training deep neural networks we have to control the size of the gradients during backpropagation. There are two prominent issues here which I won't go into in detail: vanishing gradients and exploding gradiets.

The main idea is that you want the magnitude of the gradients to remain constant as they are backpropagating through the net. A popular way to combat this is by initializing the weights of the network in smarter ways. Two initialization techniques have proven to be the most popular: Glorot initialization (sometimes referred to as Xavier initialization) and He initialization, both of which are variations of the same idea.

In order to work properly both assume that the input data is standardized. So the magnitude of the weights of your model are initialized under the assumption that your data is centered on zero and have a unit variance. It's not a good practice to feed data that is a couple of orders of magnitude larger than that.