I am training a binary classifier and at the end of every epoch I am running the trained network on the training data itself again.

Is it important to get a very high accuracy on the above step at every epoch or at least after a set of epochs?

When I did this, I saw that the negative log likelihood on the training data almost remained the same over every epoch and the recall/precision on the training data slowly was increasing. Is it possible for the negative log likelihood to remain the same but accuracy improving on the training data with every epoch?

  • $\begingroup$ What do you mean when you say that: I am running the trained network on the training data itself again? $\endgroup$ Commented Jun 21, 2015 at 16:47
  • $\begingroup$ I am passing the training data back into the network after it is trained for every epoch to check how many of the training samples does it classify correctly. It is kind of getting the training error $\endgroup$
    – London guy
    Commented Jun 21, 2015 at 20:44
  • 1
    $\begingroup$ Asking a classifier with a very high accuracy on the training set calls for overfitting. $\endgroup$ Commented Jun 21, 2015 at 23:38

2 Answers 2


Given that your network architecture is implemented correctly, and as long as they are increasing just slowly, it sounds like a learning rate problem. Probably you are using SGD with a fixed learning rate.

You should try to change the optimization method to SGD with Adagrad or Adadelta etc, that are more robust to such problems. More specifically, they are very easy to implement, you can check for more implementation details on Adagrad here: http://xcorr.net/2014/01/23/adagrad-eliminating-learning-rates-in-stochastic-gradient-descent/.

If you are using Torch7 (which is my preference), they are both implemented under the optim package. If your are using theano there are several implementations, among others: http://deeplearning.net/tutorial/code/lstm.py.

Finally, you can experiment with changing the mini-batch size.


I do also test training set on learnt network as it is a way to detect under-over fitting problem. If you can't take very high accuracy, with a non-skewed data, on training set, that means your network is under-fitting. That means it is not learning the model sufficently well becasause of number of parameters, architecture, relevance of input etc. So checking training set on learnt network is a good practice.

Negative log likelihood (NLL) is indirectly related with accuracy. Cost function is not constructed with accuracy but rather closeness measure between output probability of the network and ground truth. Accuracy is more directly related with weights. It is possible for NLL to remains almost the same while weights change considerable as in the case of plateau region. However, this behavior can be seen with updates with momentum rather than simple SGD which only depend on first order derivative of the current weights.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.