# What could be possible reason for accuracy in convnets being constant while training error keeps decreasing?

Just for sanity checks, I am checking that, can my network overfit over some 50 examples of cifar-10. But every time I see that my training accuracy is not going above 40% but training error decreases with every iteration. I am using lasagne and vgg type network. Any help for choosing hyperparameters like learning rate, momentum, regularization parameter could be bonus for me. Thanks for your attention.

• This may be a dumb question: but which accuracy are you referring to? You mean the accuracy on the training batch of size 50? Or the accuracy of your CNN on some validation/test set? – Indie AI Mar 19 '16 at 12:59
• @IndieAI fine, seems dumb question but answer it and it is training accuracy. – Siddharth Mar 20 '16 at 4:47
• You'll need to be more specific about exactly what you mean by "training error" versus "training accuracy"; they typically mean one minus the other. – Dougal Mar 20 '16 at 5:02
• @Dougal training accuracy is the fraction of correctly predicted examples from training set and training error is cost function which is cross entropy. – Siddharth Mar 20 '16 at 17:57

Call your training set $\{ (x_i, y_i) \}_{i=1}^n$, where $y_i$ is an integer from 1 to the number of classes $m = 10$; then your model predicts $\hat p_{ij}$ for each data point $i$ up to $n$ and each class $j$ up to $m$.
Your training accuracy can be written as $$\frac1n \sum_i \begin{cases}1 & y_i = \arg\max_j \hat p_{ij} \\ 0 & \text{otherwise}\end{cases}.$$ And your training loss is $$-\sum_i \log\hat p_{i y_i} .$$ So, what your network is doing is probably driving up $\hat p_{i y_i}$ for the cases it's sure about, without changing the relative ordering for the "difficult" cases. You could diagnose this further by looking at the probability outputs over time for your training set, since there are only 50 of them.