# What can I do when Overfitting doesn't seem to go away by any means?

So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model.

I wrote a neural network made without frameworks (only used numpy), and for the past two or three weeks I've been trying to train it to recognize digits and letters from a license plate dataset. The set is pretty small (~800 images), but a model with only one layer was enough to give a pretty high accuracy.

The problem is when I use it to predict on the cross validation set, as both the cost function and accuracy stay pretty much constant all the way.

The cross validation set / dev set was created by following the next steps:

• Shuffle samples,
• Split the dataset (70% training set, 30% dev set). This way I avoid having a dev set with a different distribution, as they both come from the same dataset.

This is the training set cost vs dev set cost plot, with the dataset normalized between 0 and 1 (dividing by 255):

Now this is the same plot if I normalize the images using the formula pixel = (pixel-mean)/sqrt(variance)

For the next plots I used the first normalization technique, as it gave better results. To solve overfitting, I tried:

• L2 Regularization: doesn't do much except for increasing training time, I adjusted the learning rate so you can see the overfitting.

• Dropout: I was betting on this one, as I've read it's good for computer vision and also for small datasets, but in practice training just takes a lot longer, and I don't have many parameters to try as my network is too shallow. The following plot is by training with dropout of 20% of the input layer each time forward prop is applied.

• Batch Norm: Here I'm normalizing the Z vectors inside the model, by calculating the mean and the variance and replacing each Z for (Z-mean)/sqrt(variance).

• Early stopping: doesn't work as the dev cost goes down but only like 1% or so.

• Simplifying the model: I just can't do this one, as it's as simple as it can be at the moment, the topology is [input_layer, output_layer]. The number of features is 2048 at the moment (64 by 32 px images), and the output layer has 36 units (one per each letter + the 10 numbers). I tried scaling the images down to get less features (down to 16 by 8) but the results were pretty much the same.

• Deeper network + dropout: And one last option, I added two more layers to the model (now the topology is [input_layer, 40, 40, output_layer]. The dropout percentage for each layer is [20%, 40%, 40%, 0%].

No matter what I try, it always either improves or damages learning time on the training set, but the dev set remains pretty much unaltered. I forgot to mention, as the dataset is small and I'm using vectorization, I'm using batch gradient descent, as I see no advantages to using mini batches in this case.

What am I doing wrong? Is my dataset flawed?

EDIT: So I tried the same stuff with a more standard dataset, MNIST, and I'm getting the same results. Could it be that my neural network library is broken or has a bug? I don't really know how to check because it is learning and predicting correctly, but only on the training set, so backprop and gradient descent is working as intended...

• What happens if you reduce the number of trainable parameters in the model?
– Sycorax
Jul 9, 2021 at 21:22
• @Sycorax never thought of that, but wouldn't it produce similar results to dropout? Jul 9, 2021 at 21:25
• When you split your data, do the train and test sets contain examples of all possible characters? What is the distribution in each set? What are accuracies for each character in training and in test? Is there particular characters that are the problem? Jul 9, 2021 at 21:30
• Perhaps too obvious, but are you using enough data augmentation? Jul 9, 2021 at 22:56
• @RamiroSuriano It's more like dropout is averaging over the exponentially-many thinned networks, which is a bit different than simply reducing the number of training parameters.
– Sycorax
Jul 9, 2021 at 22:59