I have read many times that a good debugging step while building a machine learning model is to try to overfit your model to a very small subset of your data. [Here is one such instance].
Provided your code is bug free, is it always possible to achieve perfect or near-perfect performance on the training set when you do this? Could you do it even on a small dataset of random numbers?
I have a model that is achieving significantly better accuracy on my actual data than it does if I feed it random numbers, but its far from perfect, and it seems no matter how small I make the dataset, how many layers I use, or how big I make the layers, the accuracy stays about the same. What could cause this?
UPDATE: Thanks to folks who responded, I understand that it should always be possible to fit a small subset of your data, so I took another look at my implementation.
It turned out there were several small issues. Switching from random uniform weight initialization to xavier initialization provided a significant bump in my results (I assumed this would only improve the speed at which training would converge to the same crappy result, but it actually improved the accuracy overall). I also did not have fully normalized data. Everything was in a range from 0 to ~10, which I initially thought should be good enough, but I got another big bump in performance when I normalized to -1 to 1. A third problem I had was with my validation set. My data is in several different sets from different sources, and it turned out there were distinct "styles" or trends to each set. I was training on a majority of the datasets, and evaluating on one particular set. When I shuffled all the individual examples together from all sets, and then drew my validation set randomly from the complete shuffled set, I started seeing accuracies in the mid and upper 90s!