The training accuracy of my model is not improving though validation accuracy improves steadily. This is weird abnormal behaviour and I just can't figure out what's wrong. Here are some graphs to help you give an idea.

Training Cost

enter image description here

Training Accuracy

enter image description here

Evaluation Accuracy

enter image description here

Training And Evaluation Accuracy Both

enter image description here

Note -- I have not used any deep learning framework. I am running this on a from scratch neural net which is taken from Michael Nielsen's book. Its working fine, I have tested it on other datasets. And the problem which I am working on is a binary Image classification task. Details are below --

Architecture - [3072, 60, 60, 2] -- two hidden layers of 60, 60 neurons
Training Data - 9000 images of dimensions (32,32,3) of two classes
Validation Data - 500 images of two classes
Activation - Sigmoid
Learning Rate - 0.005
Mini Batch Size - 10
Epochs - 35
Lambda - 0.0 (No regularization)


1 Answer 1


Assuming in 1 epoch , you go through 850 mini batches of 10 examples each..

It could be one or more of the following :-

1.) Your training/validation set is biased. Try picking validation set randomly out of training set, if you haven't .

2.) Your cost function. What is it? That too may have influenced your results..

3.) The weights that you initialized may not be good. Try using some other initial weights.

4.) You may have done some error in plotting graphs.

Try plotting the cost function for both training and validation sets. And see if their behaviour differ from the plots of accuracy.

Also, try changing batch size. Maybe 10 is too small..

  • $\begingroup$ you know what, I made some predictions using this model and submitted that to kaggle, and I got 82% accuracy. As by chance probability is 50%, it signals that model is working, at least up to some extent. I used standard cross entropy cost function and I chose training data so that it has an almost equal number of both classes, then I randomly shuffled it and selected 500 of those images for the validation set. $\endgroup$
    – Anant
    Jun 27, 2019 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.