So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model.

I wrote a neural network made without frameworks (only used numpy), and for the past two or three weeks I've been trying to train it to recognize digits and letters from a license plate dataset. The set is pretty small (~800 images), but a model with only one layer was enough to give a pretty high accuracy.

The problem is when I use it to predict on the cross validation set, as both the cost function and accuracy stay pretty much constant all the way.

The cross validation set / dev set was created by following the next steps:

  • Load whole dataset,
  • Shuffle samples,
  • Split the dataset (70% training set, 30% dev set). This way I avoid having a dev set with a different distribution, as they both come from the same dataset.

This is the training set cost vs dev set cost plot, with the dataset normalized between 0 and 1 (dividing by 255): training set cost vs dev set cost plot, dataset normalized between 0 and 1

Now this is the same plot if I normalize the images using the formula pixel = (pixel-mean)/sqrt(variance) normalized with mean and standard deviation

For the next plots I used the first normalization technique, as it gave better results. To solve overfitting, I tried:

  • L2 Regularization: doesn't do much except for increasing training time, I adjusted the learning rate so you can see the overfitting.

L2 Regularization

  • Dropout: I was betting on this one, as I've read it's good for computer vision and also for small datasets, but in practice training just takes a lot longer, and I don't have many parameters to try as my network is too shallow. The following plot is by training with dropout of 20% of the input layer each time forward prop is applied. with dropout

  • Batch Norm: Here I'm normalizing the Z vectors inside the model, by calculating the mean and the variance and replacing each Z for (Z-mean)/sqrt(variance).

with batch norm

  • Early stopping: doesn't work as the dev cost goes down but only like 1% or so.

  • Simplifying the model: I just can't do this one, as it's as simple as it can be at the moment, the topology is [input_layer, output_layer]. The number of features is 2048 at the moment (64 by 32 px images), and the output layer has 36 units (one per each letter + the 10 numbers). I tried scaling the images down to get less features (down to 16 by 8) but the results were pretty much the same.

  • Deeper network + dropout: And one last option, I added two more layers to the model (now the topology is [input_layer, 40, 40, output_layer]. The dropout percentage for each layer is [20%, 40%, 40%, 0%]. deeper network with dropout

No matter what I try, it always either improves or damages learning time on the training set, but the dev set remains pretty much unaltered. I forgot to mention, as the dataset is small and I'm using vectorization, I'm using batch gradient descent, as I see no advantages to using mini batches in this case.

What am I doing wrong? Is my dataset flawed?

EDIT: So I tried the same stuff with a more standard dataset, MNIST, and I'm getting the same results. Could it be that my neural network library is broken or has a bug? I don't really know how to check because it is learning and predicting correctly, but only on the training set, so backprop and gradient descent is working as intended...

  • 2
    $\begingroup$ What happens if you reduce the number of trainable parameters in the model? $\endgroup$
    – Sycorax
    Jul 9, 2021 at 21:22
  • $\begingroup$ @Sycorax never thought of that, but wouldn't it produce similar results to dropout? $\endgroup$
    – Ramiro
    Jul 9, 2021 at 21:25
  • $\begingroup$ When you split your data, do the train and test sets contain examples of all possible characters? What is the distribution in each set? What are accuracies for each character in training and in test? Is there particular characters that are the problem? $\endgroup$
    – bdeonovic
    Jul 9, 2021 at 21:30
  • 1
    $\begingroup$ Perhaps too obvious, but are you using enough data augmentation? $\endgroup$
    – Björn
    Jul 9, 2021 at 22:56
  • 1
    $\begingroup$ @RamiroSuriano It's more like dropout is averaging over the exponentially-many thinned networks, which is a bit different than simply reducing the number of training parameters. $\endgroup$
    – Sycorax
    Jul 9, 2021 at 22:59

1 Answer 1


I finally solved the issue a few months ago, I forgot to post here. The problem was I was merging part of the input data together by misusing the reshape method of Numpy. So in short the model was learning from basically random data, and it was impossible for it to perform well in the validation set under any circumstance.

  • 3
    $\begingroup$ 95% of data science is proper data management :) Thanks for the update. $\endgroup$
    – bdeonovic
    Oct 26, 2021 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.