This question already has an answer here:

So, While building a simpel neural net (MLP) for recognizing digits, I ordered my function to print me both the mean cost overall off the train dataset and %currect answers, also over the train dataset (even though the function doing this is called val()). the cost kept going down for an hour and a half, but the %currect answers stayed the same as the beginning. the wierd thing is that while the cosr is constantly going down, the train accurasy do not change AT ALL from the very beggining. enter image description here

Any ideas why this is happening?

Here is the nootbook, if needed.


marked as duplicate by Reinstate Monica, mdewey, kjetil b halvorsen, DeltaIV, jbowman Dec 3 '17 at 0:36

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.


It's hard to say without more detail. But most likely, if you're selecting the correct class by picking the maximal class in your softmax, then the loss can keep going down because you're increasing the correct class probability, but if another class always dominates the correct class, then the answer remains the same. For example if you have two classes with probabilities (0.2,0.8), and if the first one is supposed to be correct, then (0.3,0.7) would reduce the loss, but would not change the percent correct.

Looking at your code some more, it looks like you have three layers with 10 hidden units each. You'll most likely need at least 100 hidden units within each layer, so try making them much bigger. Check out Yann Lecun's page on MNIST accuracy here:


to compare your neural network with the baselines provided

  • $\begingroup$ this is what i thought as well. any ideas on how to fix this? this is my first neural net so im kinda lost. $\endgroup$ – מורן רזניק Dec 1 '17 at 19:23
  • $\begingroup$ @מורןרזניק: I don't see anything obviously wrong with your code, so I suggest you take a look at Yann Lecun's MNIST accuracy page to get a sense of how good or bad you're doing: yann.lecun.com/exdb/mnist Yours looks to be a (non-convolutional) neural net, so I would see if increasing the width of your layers, or adding on more layers improves the accuracy. If you want more accuracy, you'll likely have to start using convolutional neural nets. $\endgroup$ – Alex R. Dec 1 '17 at 19:56
  • $\begingroup$ @מורןרזניק: See edit $\endgroup$ – Alex R. Dec 1 '17 at 20:00
  • $\begingroup$ I followd the stracture from here:youtube.com/watch?v=aircAruvnKk&t=480s. He got a pretty great accurasy with 2 hidden layers with 16 units each. $\endgroup$ – מורן רזניק Dec 1 '17 at 20:05
  • $\begingroup$ @מורןרזניק: Could you provide a reference for the accuracy (and whether the same MNIST dataset was used)? I didn't see a mention of it. $\endgroup$ – Alex R. Dec 1 '17 at 20:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.