Low loss and low accuracy. What is the reason? We have trained two different neural networks for MNIST dataset. Here are the losses and accuracies obtained by these networks for the training data:
net0: loss: 20780.8291187
net1: loss: 209.928699374
net0: TRAIN ACCURACY     0.985890040888
net1: TRAIN ACCURACY     0.835298627336

The used loss function is the cross-entropy. We expect higher accuracies for lower losses, but here, the loss for net1 is about 100 times lower than net0 but its accuracy is lower. What is the reason?
 A: Lower cost function error not means better accuracy.
The error of the cost function represents how well your model is learning/able to learn with respect to your training examples.
Now the question is , 
Is the model learning something that I expect it to learn?
It can show very low learning curves error but when you actually testing the results (Accuracy) you are getting wrong detection's , this is called high variance.
The best is to monitor the both learning error and accuracy for each epoch/iteration, While cost function error goes down and accuracy goes up keep training, otherwise stop (:
The accuracy is not good enough? check : 
1. do I have high variance problem ? add more training examples to generalize the learning (better find more examples which reminds the problems where your model fails on).


*Do I have high bias problem ? my model is pure or to complex , need to fix it / try something else.


http://www.holehouse.org/mlclass/10_Advice_for_applying_machine_learning.html
Good luck (:
