I've been playing with TensorFlow on two similar problems, both of which are trying to classify images into one of ten classes. I'm trying a range of relatively deep convolutional network architectures (8-12 layers) inspired by the VGG net. The problems and observations of results on them are as follows:
- scaled-down problem versus full-scale problem
- 500 simple images vs 10000 slightly more complex images
- training accuracy builds to 100% vs training accuracy leaps to 65% plateau
- validation set accuracy ~85% vs validation accuracy close to training set accuracy
I'm not (yet) concerned about the 85% validation set accuracy on the simpler problem. What concerns me is how to interpret the relatively early results on the full-scale problem - particularly why the training set performance appears to leap rapidly to a low peak, and then remains essentially stuck there. The same seems to happen on all the reasonable network architectures I have tried so far. I'm using an ADAM optimiser with a learning rate of 1e-4 for the training in both cases.
What might be happening, and how might I try to fix it? Is it suggestive of getting stuck in local minimum? Or perhaps an under-estimation of the magnitude of the difference between the two problems and the naivety of using similar architectures? Or something else? What things should I be trying to get me out of such a low-performance hole?