I've been playing with TensorFlow on two similar problems, both of which are trying to classify images into one of ten classes. I'm trying a range of relatively deep convolutional network architectures (8-12 layers) inspired by the VGG net. The problems and observations of results on them are as follows:

  • scaled-down problem versus full-scale problem
  • 500 simple images vs 10000 slightly more complex images
  • training accuracy builds to 100% vs training accuracy leaps to 65% plateau
  • validation set accuracy ~85% vs validation accuracy close to training set accuracy

I'm not (yet) concerned about the 85% validation set accuracy on the simpler problem. What concerns me is how to interpret the relatively early results on the full-scale problem - particularly why the training set performance appears to leap rapidly to a low peak, and then remains essentially stuck there. The same seems to happen on all the reasonable network architectures I have tried so far. I'm using an ADAM optimiser with a learning rate of 1e-4 for the training in both cases.

What might be happening, and how might I try to fix it? Is it suggestive of getting stuck in local minimum? Or perhaps an under-estimation of the magnitude of the difference between the two problems and the naivety of using similar architectures? Or something else? What things should I be trying to get me out of such a low-performance hole?

  • $\begingroup$ did you add dropout? $\endgroup$ Commented Feb 14, 2018 at 1:08
  • $\begingroup$ Yes - very similar to VGG - layers of convolution/pooling, then fully-connected layer(s) with relu + dropout at output applied to both problems. $\endgroup$
    – omatai
    Commented Feb 14, 2018 at 1:35
  • $\begingroup$ and how long did you train for? you can expect to see things that look plateau-like during training, but over long time periods, such 'plateaus' turn out just to be learning very slowly; you can also see phenomena where the learning curve plateaus for a while, and then suddenly shifts to a new plateau; and this can occur multiple times. ADAM is pretty solid by the way. $\endgroup$ Commented Feb 14, 2018 at 1:54
  • $\begingroup$ Initially, on easier problem, I tried 10000, 5000, 2000 and 1000 epochs. Behaviour reported each 100 epochs was clear within 1000 epochs. Going beyond 2000 epochs didn't really help. On harder problem, behaviour seemed vaguely stable from epoch 200-2000. Am now experimenting with ADAM initial learning rate, and finding that higher rate improves things significantly (which is somewhat surprising, to me at least). $\endgroup$
    – omatai
    Commented Feb 14, 2018 at 2:05
  • $\begingroup$ oh right, you only have 10k images. thats pretty small... MNIST? $\endgroup$ Commented Feb 14, 2018 at 2:19

1 Answer 1


In this case, the answer turned out to have at least two components:

  • The learning rate for the ADAM optimiser was inherited from a MNIST example and set to 1e-4. Increasing it to 1e-3 (which happens to be the default) was much more appropriate for the harder of the two problems.
  • The apparent performance plateau on the harder problem was a case of learning slowly. After 50000 training iterations, I was achieving >99% training set accuracy, and approx 90% validation set accuracy.

No doubt there are further gains to be made yet.

I cannot explain why the trend in the simpler problem was one of reasonably consistent building towards 100% (e.g. 10%, 30%, 55%, 62%, ...) while the harder problem immediately appeared able to achieve 50-60% accuracy on training examples, but seemed to stay in that range for the next 2000 iterations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.