So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model.
I wrote a neural network made without frameworks (only used numpy), and for the past two or three weeks I've been trying to train it to recognize digits and letters from a license plate dataset. The set is pretty small (~800 images), but a model with only one layer was enough to give a pretty high accuracy.
The problem is when I use it to predict on the cross validation set, as both the cost function and accuracy stay pretty much constant all the way.
The cross validation set / dev set was created by following the next steps:
- Load whole dataset,
- Shuffle samples,
- Split the dataset (70% training set, 30% dev set). This way I avoid having a dev set with a different distribution, as they both come from the same dataset.
This is the training set cost vs dev set cost plot, with the dataset normalized between 0 and 1 (dividing by 255):
Now this is the same plot if I normalize the images using the formula pixel = (pixel-mean)/sqrt(variance)
For the next plots I used the first normalization technique, as it gave better results. To solve overfitting, I tried:
- L2 Regularization: doesn't do much except for increasing training time, I adjusted the learning rate so you can see the overfitting.
Dropout: I was betting on this one, as I've read it's good for computer vision and also for small datasets, but in practice training just takes a lot longer, and I don't have many parameters to try as my network is too shallow. The following plot is by training with dropout of 20% of the input layer each time forward prop is applied.
Batch Norm: Here I'm normalizing the Z vectors inside the model, by calculating the mean and the variance and replacing each Z for (Z-mean)/sqrt(variance).
Early stopping: doesn't work as the dev cost goes down but only like 1% or so.
Simplifying the model: I just can't do this one, as it's as simple as it can be at the moment, the topology is [input_layer, output_layer]. The number of features is 2048 at the moment (64 by 32 px images), and the output layer has 36 units (one per each letter + the 10 numbers). I tried scaling the images down to get less features (down to 16 by 8) but the results were pretty much the same.
Deeper network + dropout: And one last option, I added two more layers to the model (now the topology is [input_layer, 40, 40, output_layer]. The dropout percentage for each layer is [20%, 40%, 40%, 0%].
No matter what I try, it always either improves or damages learning time on the training set, but the dev set remains pretty much unaltered. I forgot to mention, as the dataset is small and I'm using vectorization, I'm using batch gradient descent, as I see no advantages to using mini batches in this case.
What am I doing wrong? Is my dataset flawed?
EDIT: So I tried the same stuff with a more standard dataset, MNIST, and I'm getting the same results. Could it be that my neural network library is broken or has a bug? I don't really know how to check because it is learning and predicting correctly, but only on the training set, so backprop and gradient descent is working as intended...
