Is a neural network required to perform with very high accuracy in the training data itself?

Question

I am training a binary classifier and at the end of every epoch I am running the trained network on the training data itself again.

Is it important to get a very high accuracy on the above step at every epoch or at least after a set of epochs?

When I did this, I saw that the negative log likelihood on the training data almost remained the same over every epoch and the recall/precision on the training data slowly was increasing. Is it possible for the negative log likelihood to remain the same but accuracy improving on the training data with every epoch?

Answer 1

Given that your network architecture is implemented correctly, and as long as they are increasing just slowly, it sounds like a learning rate problem. Probably you are using SGD with a fixed learning rate.

You should try to change the optimization method to SGD with Adagrad or Adadelta etc, that are more robust to such problems. More specifically, they are very easy to implement, you can check for more implementation details on Adagrad here: http://xcorr.net/2014/01/23/adagrad-eliminating-learning-rates-in-stochastic-gradient-descent/.

If you are using Torch7 (which is my preference), they are both implemented under the optim package. If your are using theano there are several implementations, among others: http://deeplearning.net/tutorial/code/lstm.py.

Finally, you can experiment with changing the mini-batch size.

Answer 2

I do also test training set on learnt network as it is a way to detect under-over fitting problem. If you can't take very high accuracy, with a non-skewed data, on training set, that means your network is under-fitting. That means it is not learning the model sufficently well becasause of number of parameters, architecture, relevance of input etc. So checking training set on learnt network is a good practice.

Negative log likelihood (NLL) is indirectly related with accuracy. Cost function is not constructed with accuracy but rather closeness measure between output probability of the network and ground truth. Accuracy is more directly related with weights. It is possible for NLL to remains almost the same while weights change considerable as in the case of plateau region. However, this behavior can be seen with updates with momentum rather than simple SGD which only depend on first order derivative of the current weights.