Given certain amount of labeled data, we define the net structure, such as number of layers, types of layers, the number of convolutional layers, the number of pooling layers, etc.

And train the parameters using back propagation, meanwhile we show the loss in training procedure and view the testing accuracy in validating data set.

But, the loss in training set is nearly zero, and the testing accuracy is kept unchanged no matter how to decrease the learning rate.

  • In this circumstance, is it overfitting?
  • Should we change the net structure?
  • More layers for more parameters?
  • Could you please recommend some suggestions or references?

2 Answers 2


I would say there might be a bug when calculating the errors when using backpropagation.

Also how big is your data? And how many epochs are you using for your simulation? How is your training being done? I could help you more if i could get more data.

This might help you: http://www.cs.toronto.edu/~hinton/csc2515/notes/lec4.htm

  • $\begingroup$ Thank you for replying, sir! I've finally been told that indeed the amount of the training data is very important to the CNN. Thus, a common approach is to preprocess the limited data samples to simulate more data samples. But now I have no much time to do that work. Moreover, the batch_size of SGD also has effects in the training procedure, larger the batch_size, smoother the training loss. Additionally, I mainly use the open source caffe to train the CNN. But, I'm not sure that what's the bug of backpropagation as you have presented. Could you please give more details? Thanks! $\endgroup$
    – mining
    Aug 3, 2014 at 13:41

Maybe you could try the dropout technique. I have heard it can be effective against overfitting.

  • Dropout: A simple way to prevent neural networks from overfitting, by Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov Journal of Machine Learning Research, 2014. PDF

Recently, Google even patended this technique!!!

  • 2
    $\begingroup$ Welcome to the site. We strive to build a permanent repository of high-quality statistical information. Thus, we worry about linkrot. Can you provide a complete citation for the paper, & a summary of the information there / how it can resolve the OP's problem? $\endgroup$ Mar 19, 2015 at 19:17
  • $\begingroup$ Here is the patent: google.de/patents/… $\endgroup$ Mar 1, 2017 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.