Rule of thumb Overfit in a MLP or is it possible with N = 135

Question

I want to say ahead that I highly appreciate any Literature recommendation (Book, blogg, ...) besides ESL and ISL.

My Question: Is it possible to train a 3 layer multilayer perceptron (mlp) in a binary classification problem with just 135 observations with out massive over fitting?

I know this is a complex topic and the required size depends on many factors, and there is most certainly not just one right answer.

However I read as a rule of thumb: At least 10 observations / per parameter (in case of mlp: weights), can somebody confirm this rule?

Details to the data:
N = 135
M = 20 Features = 20 Input Neurons
J = Neuron ins hidden layer = ??
Y / O = binary class = 2 Neurons

Parameters/Weights So for instance if we assume just 5 neurons in the hidden (J = 5, which is imo probably to low), this would mean we are searching for 110 Paramters

M*J + J * O = 20*5 + 5*2 = 110

When I would apply the 10 obs / parameter rule, it would be impossible to prevent overfitting (110 params * 10 obs = 1100 obs would be required).

I had to ask this, because I already know that this will wander my mind and steal my sleep.

Best wishes

Answer 1

Is it possible to train a 3 layer multilayer perceptron (mlp) in a binary classification problem with just 135 observations with out massive over fitting?

Yes, it is.

At least 10 observations / per parameter (in case of mlp: weights), can somebody confirm this rule?

No, that's not so simple. While I appreciate such heuristics I also take that for what they are, heuristics. If you have more degrees of freedom (basically flexibility) in your model than you have samples then it can, theoretically, memorize the data, which might lead to catastrophic overfitting (not always though).

Thing is, the number of parameters is not exactly the same as the degrees of freedom of a model.

You can artificially induce constrain your parameters. This is called regularization (see our questions and answers on it regularization). On MLPs, regularization can be used explicitly (you want your model to reduce a mis-classification cost plus a sum of the absolute values of the weights, i.e. $\ell$-1 regularization, so it has less freedom on its selection of weights), but there are other implicitly defined regularizations (for example, batch learning).

The biggest artificial neural networks are in an overparametrized regime, where they have way more parameters than the number of samples they'll ever be exposed to. See the top performing networks on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) for example. An early top performer, VGG-19 has 143,667,240 trainable parameters. ImageNet has less than 1 million samples.