Validation Accuracy Increases But Training Accuracy Doesn't

Question

The training accuracy of my model is not improving though validation accuracy improves steadily. This is weird abnormal behaviour and I just can't figure out what's wrong. Here are some graphs to help you give an idea.

Training Cost

Training Accuracy

Evaluation Accuracy

Training And Evaluation Accuracy Both

Note -- I have not used any deep learning framework. I am running this on a from scratch neural net which is taken from Michael Nielsen's book. Its working fine, I have tested it on other datasets. And the problem which I am working on is a binary Image classification task. Details are below --

Architecture - [3072, 60, 60, 2] -- two hidden layers of 60, 60 neurons
Training Data - 9000 images of dimensions (32,32,3) of two classes
Validation Data - 500 images of two classes
Activation - Sigmoid
Learning Rate - 0.005
Mini Batch Size - 10
Epochs - 35
Lambda - 0.0 (No regularization)

Answer 1

Assuming in 1 epoch , you go through 850 mini batches of 10 examples each..

It could be one or more of the following :-

1.) Your training/validation set is biased. Try picking validation set randomly out of training set, if you haven't .

2.) Your cost function. What is it? That too may have influenced your results..

3.) The weights that you initialized may not be good. Try using some other initial weights.

4.) You may have done some error in plotting graphs.

Try plotting the cost function for both training and validation sets. And see if their behaviour differ from the plots of accuracy.

Also, try changing batch size. Maybe 10 is too small..